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The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(ARl)  Is  Interested  in  developing  a  Human  Reliability  Growth  Model  (HRGM)  that 
accurately  predicts  relationships  between  relevant  training  variables  and 
soldier  task  performance.  Of  specific  Interest  would  be  a  model  that  imple¬ 
ments  data  related  to  the  Initial  training  of  soldiers  at  the  U.S.  Army  Train¬ 
ing  and  Doctrine  Command  (TRADOC)  schools  and  the  effect  of  training  time  on 
soldier  performance  of  military  tasks. 

Approaches  to  this  problem  are  typified  by  methods  that  are  highly 
qualitative  in  nature  (e.g.i  anecdotal  observations).  However,  these  ap¬ 
proaches  result  in  no  direct,  quantitative  evaluation  of  the  Impact  of  train¬ 
ing  on  soldier  task  performance.  An  approach  that  combines  the  benefits  of 
existing  qualitative  data  collection  techniques  with  the  precision  of  quanti¬ 
tative  data  would  be  most  useful  to  the  Army, 

This  report  examines  the  feasibility  of  developing  an  HRGM  that  predicts 
the  impact  of  training  variables  on  soldier  performance.  Such  a  model  would 
be  configured  to  be  both  stand  alone  (i.e.,  requiring  no  additional  analytic 
support  for  the  performance  of  analyses  using  the  model)  and  quantitative.  In 
Older  to  satisfy  these  criteria,  the  HRGM  would  have  to  be  based  on  empirical 
data  obtained  from  the  behavioral  and  social  science  literature,  as  well  as 
available  government  data  bases. 

Thus,  the  primary  focus  of  this  feasibility  study  was  to  examine  whether 
empirical  data  could  be  collected  to  support  the  development  of  an  HRGM.  As 
an  additional  part  of  the  conceptualization  of  the  HRGM,  research  underlying 
human  learning  curves  and  curve  fitting  techniques  was  reviewed  to  predict  the 
impact  of  training  on  soldier  pv^rformance  based  on  learning  curve 
relationships . 

Results  of  this  study  were  used  to  evaluate  the  feasibility  of  develop¬ 
ing  a  stand  alone  HRGM  based  on  quantitative  relationships  between  numerous 
training  variables  and  soldier  performance.  Moreover,  results  from  this  study 
may  serve  as  a  basis  for  future  models  or  al<^orlthms  requiring  such 
literature. 
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FEASIBILITY  STUDY  FOR  PRBDICTIN6  HUMAN  RELIABILITY  GROWTH 
THROUGH  TRAINING  AND  PRACTICE 

Introduction 


The  Army  Research  Institute  (ARI)  requires  the  development 
of  a  Human  Reliability  Growth  Model  (HRGM)  to  predict  the 
relationship  of  numerous  training  variables  on  soldier  task 
performance.  Of  particular  interest  is  a  focus  on  (a)  initial 
training  of  soldiers  as  would  occur  at  TRADOC  (U.S.  Army  Training 
and  Doctrine  Command)  schools,  (b)  effect  of  training  time  (time 
to  learn)  on  soldier  performance  of  military  taslcs,  and  (c) 
empirically  derived  relationships  between  training  and 
performance.  Such  a  model  should  be  configured  to  be  both  stand 
alone  (i.e.,  requiring  no  additional  analytic  support  for  the 
performance  of  analyses  using  the  model)  and  quantitative.  The 
development  of  such  a  model  has  to  be  based  on  empirical  data 
obtained  from  the  behavioral  and  social  science  literature  and 
available  government  data  bases.  Thus,  the  purpose  of  this 
project  was  to  examine  the  feasibility  of  developing  this  model 
given  the  risks  attendant  to  collecting  the  sort  of  empirical 
data  (i.e.,  the  data  may  not  be  available  or  may  not  exist) 
required  to  successfully  develop  such  a  model. 

The  literature  on  learning  curve  fitting  techniques  was 
reviewed  as  part  of  the  feasibility  study.  Incorporating  current 
views  and  research  underlying  human  learning  curves,  a  model  was 
conceptualized  based  on  relationships  defined  through  learning 
curve  fitting  techniques.  Specifically,  the  Growth  Model  would 
predict  the  effect  of  training  on  performance  based  on 
consideration  of  human  (i.e.,  soldier)  capabilities  and  potential 
improvements  in  performance  due  to  training.  The  characteristics 
of  a  HRGM  would  be 

•  Provides  the  capability  to  predict  the  impact  of 
training  variables  (e.g.,  time,  media,  and  method)  on 
soldier  performance  based  on  learning  curve 
relationships 

•  Provides  formalized  quantitative  relationships  of  the 
impact  of  training  on  human  performance  based  on 
empirical  data  obtained  from  actuarial  data  sources 

•  Ma}ces  use  of  taxa-based  (i.e.,  tas)c-based) 
relationships  to  order  and  organize  the  impact  of 
training  on  performance. 

This  feasibility  study  became  an  effort  to  address  the  following 
question  as  a  primary  focus  of  the  research  effort; 


1 


Do  sufficient  amounts  of  suitable  data  exist? 


In  order  to  address  this  question,  two  parallel  activities 
were  pursued.  The  first  was  an  effort  to  collect  all  the 
available  literature  that  presents  empirical  data  on  the  effects 
of  human  learning  and  practice  on  human  performance.  This 
activity  consisted  of  (1)  literature  searches  of  on-line  data 
bases  of  human  behavioral  and  social  science  literature 
abstracts,  (2)  personal  contacts  with  government  points-of- 
contact  for  sources  of  government  owned  data  bases  containing  the 
required  empirical  data,  (3)  'brute-force'  investigation  of 
personal  and  corporate  research  libraries  to  identify  relevant 
literature,  and  (4)  limited  direct  contact  with  current 
researchers  known  to  have  conducted  research  in  these  areas.  The 
research  collected  through  this  manner  was  reviewed,  archived, 
abstracted,  and  logged  in  a  data  base  for  later  use.  Out  of 
approximately  3,000  research  titles  and  abstracts  reviewed,  27 
research  articles  were  selected  as  meeting  minimal  criteria  for 
acceptable  utility. 

The  second  activity  was  a  comprehensive  review  of  the 
theoretical  literature  involving  human  learning  and  practice, 
with  particular  focus  on  human  learning  curve  fitting  techniques. 
This  review  encompassed  70  years  of  research  beginning  with 
Thurstone  (1919),  in  which  the  nature  and  application  of  learning 
curves  were  derived  and  summarized.  The  review  also  helped  in 
providing  focus  to  the  search  and  collection  of  empirical 
learning  and  performance  data.  A  final  step  was  the  development 
of  criteria  for  determining  the  acceptability  of  empirical  data 
for  use  in  developing  an  HRGM  that  meets  the  project  criteria  for 
being  stand  alone  and  quantitative. 

The  final  activity  in  this  project  was  to  evaluate  the  data 
collected  with  respect  to  the  criteria  established  for  their 
acceptance.  This  analysis  was  used  to  determine  the  feasibility 
of  developing  a  successful  growth  model,  given  project 
constraints . 

This  report  contains  a  technical  section  in  addition  to  the 
introduction.  The  first  part  of  the  technical  section  discusses 
the  theoretical  bases  for  the  measurement  and  prediction  of  human 
learning  and  practice  and  performance.  The  second  part  presents 
a  detailed  description  of  the  empirical  data  collection  effort 
and  a  summary  of  the  results  obtained.  The  third  part  presents  a 
discussion  of  the  overall  feasibility  of  developing  a  successful 
HRGM  that  meets  project  requirements. 

This  report  also  contains  a  section  of  five  appendixes. 
Appendix  A  presents  the  empirical  data  and  research  abstracts 
from  the  27  articles  that  met  minimal  criteria  for  developing 
the  HRGM,  and  Appendix  B  contains  references  for  the  27  articles. 
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Appendix  C  is  a  bibliography  of  all  articles  reviewed  for 
inclusion  in  the  HRGM.  Appendix  0  presents  detailed  algebraic 
derivations  of  various  learning  curve  functions  reported  in  the 
literature.  Finally,  Appendix  E  contains  a  bibliography  of 
supplemental  research  that  is  relevant  to  the  project,  but  not 
central  to  the  effort. 
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Thttoratlcftl  Baaas  for  th«  Maasuromont  and  Pradictlon 
of  Human  Laarning  and  Praotioa 


The  relation  between  training  variables  and  soldier 
performance  must  be  determined  in  order  to  develop  a  stand  alone 
HRGM.  The  most  pervasive  approach  used  by  learning  researchers 
to  assess  the  relationship  between  training  (i.e.,  learning  and 
practice)  and  performance  <e.g.,  time  or  trials)  is  curve 
fitting.  Essentially,  curve  fitting  describes  changes  in 
performance  over  time  or  trials  using  the  typical  "acquisition" 
curve  prevalent  in  studies  of  skill  learning  and  retention  (Lane, 
1986b)  . 

The  purpose  of  this  chapter  is  to  review  the  literature  on 
the  measurement  and  prediction  of  the  major  aspects  of  training 
and  performance.  First,  this  encompasses  the  behavioral  science 
literature  involving  the  learning  curve  and  its  relation  to  the 
measurement  and  prediction  of  the  effects  of  human  learning  and 
practice  on  the  attainment  of  skilled  performance.  Second,  the 
behavioral  science  literature  on  the  measurement  and  prediction 
of  the  effects  of  training  variables  on  the  performance  of 
military  tasks  are  reviewed.  Finally,  the  requirements  for  a 
HRGM  and  associated  data  requirements  are  identified. 

Theoretical  Roots  of  the  Learning  Curve  and  Its  Heuristics 

Since  Thurstone  (1919)  laid  out  his  formal  definition  of  the 
learning  curve;  the  "amount  of  attainment  gained  per  unit  of 
practice  decreases  as  practice  increases",  the  'ubiquitous'^ 
learning  curve  has  come  to  dominate  the  area  of  human  learning 
and  practice  as  the  metric  of  choice  among  behavioral  scientists. 
While  not  the  first  to  develop  such  constructs,  Thurstone  derived 
the  prototypical  equation  for  fitting  cumulative  errors  (or 
conversely  cumulative  successes)  with  a  hyperbolic  function  or 
curve  (Lane,  1986b) He  identified  the  notable  characteristics 
of  the  learning  curve  which  over  time  have  more  or  less  held 
constant  across  areas  of  human  learning  research.  In  addition  to 
the  description  of  the  general  appearance  of  the  learning  curve 
noted  above,  the  more  notable  characteristics  among  those 
Thurstone  identified  include: 


^The  word  'ubiquitous'  has  been  employed  by  a  number  of  more  recent  learning 
researchers  (e.g.,  Newell  &  Rosenbloom,  1981)  as  a  description  of  the  persistence  of  the 
learning  curve  as  the  metric  of  choice  and  because  few  other  operational  learning 
constructs  have  evolved  as  alternatives. 


*  For  the  sake  of  simplicity  and  to  avoid  the  ongoing  debate  as  to  the  optimum  form 
of  the  learning  curve  function,  the  major  learning  curve  equations  (as  well  as  their 
derivations)  that  have  been  discussed  in  the  literature  over  the  years,  are  available  in 
Appendix  D  of  this  report. 
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•  Learning  curves  sometimes  take  the  form  of  a  "positive 
acceleration  at  the  initial  stage  of  learning,  plateaus 
during  the  course  of  learning,  and  erratic  advancement 
of  attainment . " 

•  "The  use  of  an  equation  for  this  relation"  (i.e.,  the 
attainment  of  skills  over  trials  or  practice  time) 
"enables  one  to  predict  the  limit  of  practice  before  it 
has  been  attained,  provided  that  the  learning  follows 
the  law  of  diminishing  returns." 

•  Rate  of  learning  can  be  differentiated  from  the  limit 
of  practice  (i.e.,  asymptote)  since  these  are 
"undoubtably  independent"  metrics. 

•  The  learning  curve  equation  can  "be  of  service  in  the 
analysis  of  the  relation  between  the  variability  in 
learning  and  other  mental  attributes." 

•  "All  questions  of  transfer  of  training  may  be 
investigated  by  the  learning  equation  and  the  transfer 
effect  may  be  differentiated  into  psychological 
components . " 

In  1930,  Thurstone  revisited  his  earlier  effort, 
strengthened  the  premises  he  postulated  by  adding  new  inferences 
to  them,  and  evolved  more  complex  forms  of  his  learning  curve 
equations.  These  additional  characteristics  of  the  learning 
curve  include; 

•  "The  two  principal  variables  in  learning  are:  (a) 
practice,  and  (b)  attainment.  Attainment"  is  "an 
increasing  function  of  practice.  Practice  may  be 
measured  either  in  terms  of  repetition  or  in  terms  of 
time  devoted  to  it,  and  effort  is  assumed  to  be 
constant.  Repetitions  may  be  counted  as  successes  and 
failures . " 

•  "Learning  consists  in  a  series  of  separable  acts,  that 
some  of  these  are  counted  as  successful  and  that  others 
are  counted  as  errors.  Each  act  whether  right  or  wrong 
will  be  counted  as  a  unit  of  practice." 

•  "Attainment"  (also  expressible  in  various  ways)  can  be 
expressed  as  a  measure  of  "the  probability  that  an  act 
will  be  counted  as  successful.  This  measure  of 
attainment  is  proportional  to  and  consequently  for  our 
purposes  synonymous  with  the  number  of  successful  acts 
per  unit  time." 

In  a  pair  of  articles,  Thurstone  (1919;  1930)  identified  the 
major  learning  curve  parameters  and  the  typical  behavior  (or 
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form)  that  the  learning  curve  equations  often  take.  These 
parameters  and  the  behavior  (form)  of  the  learning  curve  have 
dominated  research  up  to  the  present  time.  Thurstone  identifies: 


•  the  major  learning  curve  parameters: 

(a)  rate  of  learning 

(b)  asymptote  ('limit  of  practice') 

(c)  initial  learning  rate  ('positive  acceleration  at 
the  initial  stage  of  learning' ) , 

•  the  hyperbolic  form  of  the  learning  curve  equation, 

•  the  rules  for  the  measurement  and  prediction  of 
learning  and  practice  effects  on  skill  attainment, 
including: 

(a)  assumptions  underlying  the  methodology  for  the 
measurement  of  practice, 

(b)  assumptions  underlying  the  measurement  of 
successes  or  failures  to  demonstrate  an  operation, 
and 

(c)  approximate  equivalence  of  'the  probability  that 
an  act  will  be  counted  as  successful'  and  'the 
number  of  successful  acts  per  unit  time' ,  and 

•  the  practical  implications  of  measuring  and  predicting 
human  learning  and  practice  effects  on  human 
performance : 

(a)  transfer  of  training  and  transfer  effects 

(b)  relation  of  human  learning  and  practice  to  other 
human  mental  attributes. 

Each  of  these  characteristics  (and  others  added  by  later 
researchers)  will  be  discussed  in  a  section  for  determining  the 
requirements  for  a  HRGM  and  the  data  requirements  associated  with 
such  a  model. 

Families  of  Learning  Curves 

Since  Thurstone  (1919/  1930),  several  additional  functions 
or  families  of  learning  curves  have  been  identified  based  on 
regularities  in  the  shapes  of  the  curves. 

Power  law.  One  of  the  most  prevalent  learning  curves 
identified  is  the  power  law  of  practice  or  power  function  (Lane, 
1986b;  Newell  &  Rosenbloom,  1981) .  The  power  function  fits  a 
wide  variety  of  skill  data  such  as  perceptual-motor  skills,  motor 
skills,  and  cognitive  skills,  and  is  "ubiquitous  when  one 
measures  the  log  of  performance  time  against  the  log  of  trial 
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number"  (Newell  &  Rosenbloom,  1981,  p.  2) .  Its  shape  Is 
negatively  accelerated.  That  is,  the  gain  in  performance  on  a 
trial  decreases  as  practice  increases  (see  Figure  1,  Curve 
1-A) 


The  power  function  takes  the  form; 


T  =  A  +  B  (N  +  E)'“ 


where 

T  =  time  to  respond  or  to  perform  a  task  (completion 
of  one  unit  of  output  on  a  given  trial) , 

A  =  asymptote  or  highest  level  of  performance 
attainable, 

B  =  performance  or  output  on  first  trial, 

N  =  number  of  trials  or  time  units  (index  of 
practice) , 

E  =  prior  learning  (transfer  from  prior  experience  or 
learning  required  to  attain  entry  level 
performance) , 

R  =  rate  variable  (amount  of  change  in  Y  or  T  with  one 
unit  change  in  N  or  average  slope  of  the  curve) . 

In  power  functions,  the  independent  and  dependent  variables 
are  usually  transformed  logarithmically,  and  this  log-log 
transformation  yields  a  relatively  straight  line  (Lane,  1986b; 
Newell  &  Rosenbloom,  1981)  . 

Hyperbolic .  The  hyperbolic  function  (introduced  above 
during  the  discussion  of  Thurstone's  research)  is  derived  from 
the  power  function  but  requires  the  fitting  of  one  less  parameter 
to  the  data  because  it  assumes  an  implied  rate  variable  (R)  of  -1 
(Lane,  1986b) . 


’it  should  be  noted  that  the  families  of  curves  depicted  in  Figure  1  are  generalized 
to  show  their  differences. 
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Figure  1.  General  shapes  of  power,  exponential,  hyperbolic,  and 
logistic  curves. 

The  hyperbolic  function  is; 

T^A  +  B/  (N  +  E) 

where 

T  -  time  to  respond  or  perform  a  task, 

A  =  asymptote 

B  =  performance  or  output  on  first  trial 
N  =  number  of  trials 
E  =  prior  learning. 

When  B  is  positive,  the  function  will  be  decreasing,  whereas 
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the  function  will  be  increasing  when  B  is  negative.  A 
generalized  hyperbolic  curve  is  shown  in  Figure  1,  Curve  1-C. 

Exponential .  The  third  family  of  curves  found  in  learning 
data  is  the  exponential  function.  The  exponential  function 
differs  from  the  power  function  in  that  it  is  semi-logarithmic; 
only  one  variable  is  transformed  logarithmically.  In  addition, 
the  exponential  curve  decreases  and  increases  in  a  more  rapid 
fashion  than  the  power  curve  because  the  amount  learned  on  each 
trial  does  not  decrease  as  a  function  of  N  (the  number  of 
trials) .  Thus,  the  exponential  curve  "produces  a  curve  of 
constant  acceleration,  producing  a  much  'steeper'  curve  than  the 
power  law"  (Lane,  1986b,  p.  30) .  An  example  of  a  generalized 
exponential  curve  is  shown  in  Figure  1,  Curve  1-B. 

The  form  of  the  exponential  function  is: 

T  =  A  +  B  e 


where 

T  =  time  to  respond  or  perform  a  tas)c, 

A  =  asymptote 

B  =*  performance  or  output  on  first  trial 
e  =  natural  logarithm 
R  =  rate  variable 
N  =  number  of  trials 


Logistic .  The  final  member  in  the  family  of  curves  is  the 
logistic  function.  The  logistic  curve  is  not  as  common  in 
learning  data  as  power  or  exponential  curves,  and  appears  to  be 
derived  primarily  from  data  involving  performance  ratings  by 
instructors  in  training  or  assessment  situations  (e.g.,  check 
pilot  evaluations  performed  in  the  aviation  training  community 
prior  to,  during,  and  following  formal  training  of  student 
pilots) .  The  logistic  function  is  an  "S"  shaped  or  sigmoid  curve 
that  is  both  positively  and  negatively  accelerated.  In  other 
words,  the  curve  rises  slowly  in  early  trials,  accelerates 
rapidly  in  the  middle  trials,  and  levels  off  in  later  trials  (see 
Figure  1,  Curve  ID) .  As  seen  in  Figure  2,  the  logistic  curve  may 
contain  two  inflection  points  (i.e.,  the  point  in  a  curve  where 
positive  acceleration  changes  to  negative  acceleration  or  where 
negative  acceleration  changes  to  positive  acceleration)  compared 
to  power  and  exponential  curves  that  contain  a  single  inflection 
point  (Lane,  1986b) . 
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The  logistic  function  takes  the  form; 


Y  =  A  /  [1  +  <B  -  A)  e 


where 

Y  =  performance  score,  ratings,  or  any  other  measure 
that  is  represented  by  increasing  values 

A  =  asymptote 

B  *  performance  or  output  on  first  trial 

e  =  natural  logarithm 

k  =  an  implicit  function  of  R,  representing  the 

percentage  of  learning  to  be  accomplished  on  each 
trial;  k  =  R  /  [Y  (A  -  Y) ] 

N  =  number  of  trials 


The  logistic  function  is  often  referred  to  as  the 
"autocatalytic  equation”  because  previous  learning  "catalyzes" 
subsequent  learning.  That  is,  previous  learning  speeds  up 
subsequent  learning  resulting  in  a  "snowballing"  effect  (Lane, 
1986b;  Spears,  1983) . 

This  discussion  has  sought  to  focus  on  the  major  topics  and 
parameters  identified  in  the  literature  on  human  learning  curves 
and  their  mathematical  forms.  We  have  sought  to  avoid  the 
controversy  regarding  the  "correct"  form  of  the  learning  curve 
function  extant  in  the  available  literature.  Rather,  we  believe 
that  it  is  possible  that  a  number  of  these  curves  are  on 
approximately  equal  footing  with  respect  to  the  legitimacy  of 
their  use  for  any  given  task  and  training  situation.  As  a 
consequence,  the  selection  of  a  learning  curve  function  should  be 
based  on  empirical  data  specific  to  the  training  context  and 
nature  of  the  task.  Later  researchers  have  identified  some  of 
the  parameters,  conditions,  and  constraints  on  the  use  of 
learning  and  performance  data.  This  body  of  research  is  the 
focus  of  the  next  section  of  this  chapter. 

Applications  of  Learning  Curves  to  Military  Training 

Recently,  Spears  (1983)  has  used  learning  curve  theory  to 
explain  transfer  of  training  (TOT) .  Although  Spears  explains  TOT 
within  the  context  of  training  devices  (particularly  flight 
training  using  simulators) ,  his  principles  are  generalizable  to  a 
large  variety  of  training  situations  and  tasks.  According  to 
Spears,  the  transfer  of  skills  from  training  (such  as  on  a 
training  device)  to  the  operational  environment  (such  as  to 
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Figure  2.  Logistic  curve  with  two  inflection  points. 


operational  equipment)  is  essential  in  training.  Therefore,  it 
is  important  to  recognize  and  understand  the  empirical  indicators 
or  parameters  of  learning  curves  on  which  researchers,  such  as 
Spears,  focus  during  the  measurement  and  evaluation  of  training. 

Spears  describes  four  basic  parameters  of  learning  curves 
that  are  important  for  the  study  of  learning  and  transfer  in 
applied  settings: 

a)  beginning  level, 

b)  asymptote, 

c)  rate  of  learning,  and 

d)  inflection  point. 

The  first  parameter,  beginning  level,  refers  to  the  level  of 
performance  on  a  task  before  any  practice  has  occurred. 

According  to  Spears  (1983),  examining  the  beginning  level  of 
curves  may  help  in  comparisons  of  different  methods  of  training. 
For  example,  comparing  the  beginning  levels  of  different  methods 


of  training  and  transfer  may  help  to  determine  which  method  would 
be  more  cost  effective.  Spears  cautions,  however,  that  beginning 
level  performance  is  notoriously  unstable  when  applied  to  the 
context  of  predicting  later  learning  based  on  initial  learning. 
For  this  reason,  he  limits  the  application  of  beginning  level  as 
a  parameter  and  cautions  researchers  to  collect  data  on  a 
sufficient  number  of  trials  to  avoid  the  instability. 

The  second  parameter,  asymptote,  is  the  highest  level  of 
performance  attainable  for  all  practical  purposes.^  It 
indicates  what  is  or  could  be  accomplished  during  training. 
Asymptote  is  important  as  an  indicator  of  the  point  at  which  a 
given  stage  of  skill  integration  is  completed  and  designates  the 
level  of  achievement  at  that  stage.  Asymptote  may  also  be  useful 
when  comparing  skill  integration  between  two  methods  of  training 
(i.e.,  one  method  requires  less  training  to  reach  asymptote  than 
another) .  It  is  most  useful  for  comparisons  of  training  time, 
devices,  and  methods. 

The  third  parameter,  rate  of  learning,  is  the  measure  used 
to  assess  training  effectiveness  (the  change  in  achievement  per 
unit  of  change  in  the  indepei  .it  variable)  .  It  is  a  primary 
parameter  for  enabling  decisions  regarding  training  resource 
allocations  (e.g.,  time,  materials,  and  people)  and  amount  of 
practice . 

The  last  parameter,  inflection  point,  is  the  point  in  a 
curve  where  positive  acceleration  changes  to  negative 
acceleration  or  vice-versa.  Examining  the  inflection  point (s)  of 
learning  curves  may  be  important  when  examining  the  integration 
of  multi -component  skills;  they  could  reveal  when  integrative 
processes  are  complete.  Inflection  points  could  also  indicate 
when  a  level  of  proficiency  is  reached  in  a  transfer  task. 
However,  there  have  been  no  practical  applications  of  using 
inflection  points  to  study  learning  and  transfer  to  date. 

Clearly,  any  of  these  four  basic  parameters  are  potentially 
useful  to  address  specified  training  concerns.  They  should  be 
considered  for  potential  application  within  the  context  of  a 
HRGM. 


Other  researchers,  notably  Lane  (1986b),  have  used  learning 
curve  theory  to  describe  the  general  influence  of  key  variables 
on  military  training  and  education.  Initially,  Lane  attempted  to 
develop  a  generalized  learning  curve  which  was  to  be  used  to 
estimate  the  duration  of  training.  However,  he  found  that  the 
published  literature  would  not  support  the  development  of  a 


^However,  attainment  of  skilled  performance  can  continue  to  a  significant  degree  many 
thousands  of  practices  or  trials  beyond  what  many  researchers  would  designate  as 
asymptote . 
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quantitative  prediction  of  training  time.  He  cites  the  following 
as  explanation  for  the  failure  to  develop  a  generalized  learning 
curve ; 

•  the  literature  on  learning  curves  did  not  contain  the 
type  of  quantitative  data  needed  to  estimate  training 
time, 

•  key  information  was  often  omitted  in  published  studies, 

•  many  of  the  tasks  studied  were  too  simple, 

•  many  tasks  were  not  representative  of  military  tasks, 

•  many  of  the  curves  reported  in  the  literature  were 

group  curves  (based  on  group  data) ,  and 

•  curves  based  on  individual  performance  data,  when 
reported,  differed  from  the  group  curves. 

Therefore,  Lane  (1966b)  dropped  the  effort  to  develop  a 
generalized  learning  curve  and  instead  focused  his  efforts  on  the 
description  of  conditions  affecting  acquisition  curves  that  can 
be  applied  to  military  tasks.  The  conditions  he  identified  which 
impact  learning  acquisition  curves  are  summarized  below, 

1 .  The  learning  of  multi-component  tasks  (common  in 
military  training)  may  cause  temporary  asymptotes  or 
"plateaus”  in  acquisition  curves,  These  temporary 
plateaus  may  be  due  to  shifts  in  the  focus  on  varying 
components  of  the  task  being  learned.  Over  the 
training  period,  trainees  may  shift  their  attention 
from  easier  components  (mastered  early)  to  other  more 
complex  components  (mastered  later) .  These  shifts  may 
be  reflected  in  the  occurrence  of  temporary  plateaus  in 
skill  attainment.  The  integration  of  components  -  as 
described  by  Spears  (1983)  in  the  process  of  skill 
integration  -  may  also  play  a  role  in  the  occurrence  of 
temporary  plateaus. 

2 .  Multi-component  or  complex  tasks  may  cause  a  great  deal 
of  variation  in  individual  acquisition  curves. 

3.  Differences  in  ability  may  affect  the  shape  of  the 
acquisition  curve.  Both  asymptote  and  rate  of  learning 
may  vary  in  individuals  learning  the  same  task. 

4 .  The  degree  of  prior  learning  and  different  training 
methods  may  affect  the  shape  of  the  acquisition  curve. 


According  to  Lane  (1986b),  the  study  of  learning  curves  is 
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important  because  it  can  reveal  ultimate  skill  proficiency, 
determine  when  sufficient  training  or  practice  has  been  reached, 
and  track  an  individual's  performance  or  progress.  His 
conclusions  that  are  relevant  to  this  discussion  are  summarized 
below. 

1.  "The  'typical'  curve  relating  training  performance  to 
practice  has  a  characteristic  negatively  accelerated 
shape."  Curves  deviate  frequently  from  that  shape  for 
a  variety  of  reasons  and  due  to  differing  conditions 
(discussed  above) . 

2.  While  the  general  shape  of  the  curve  can  be  reliably 
anticipated,  the  "time  course"  over  which  acquisition 
runs  -  and  as  a  consequence-  its  parameters  cannot  be 
reliably  predicted  from  prior  knowledge  of  task 
characteristics  alone.  The  mathematical  description  of 
a  task  learning  curve  requires  data  specific  to  a  task 
or  training  segment. 

We  share  some  of  the  concerns  raised  by  Lane  (1986b), 
particularly  those  which: 

(a)  relate  to  constraints  on  military  training  (e.g.,  the 
need  to  train  everyone  in  fixed  time  intervals  tailored 
more  to  economics  than  to  the  attainment  of  skilled 
performance) , 

(b)  confusion  in  the  field  regarding  the  application  and 
meaning  of  learning  curves  based  on  individual  data 
versus  group  data,  and 

(c)  the  masking  effects  which  tend  to  occur  in  the  context 
of  multi-component  tasks. 


As  a  consequence,  a  HRGM  should  take  these  issues  into 
consideration.  Constraints  on  military  training,  however,  appear 
to  have  more  to  do  with  the  judicious  use  of  the  growth  model 
than  upon  any  model-intrinsic  factors.  The  question  of  learning 
curves  based  on  individual  versus  group  data  can  be  managed 
within  the  context  of  alternatives  built  into  the  growth  model. 
For  instance,  users  of  the  model  should  be  cautioned  against  the 
inappropriate  application  of  curves  based  on  group  data  when  only 
individual  data  apply.  Group  curves  have  their  place  in  the 
growth  model  since  many  projections  of  interest  would  be  relevant 
to  trade-offs  among  varying  levels  of  soldier  aptitude  (e.g., 

AFQT  -  Armed  Forces  Qualification  Test  -  scores  and  mental 
category) .  Finally,  the  issue  of  multi -component  tasks  can  only 
be  addressed  once  the  growth  model  for  single-component  tasks  has 
been  developed.  This  issue  would  require  additional  research  to 
determine  the  effects  of  multi -component  tasks  on  the  shape  of 
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learning  curves. 

Other  Approaches  to  Predicting  Human  Performance 

In  addition  to  curve  fitting  techniques,  some  researchers 
have  attempted  to  predict  human  performance  using  models .  Human 
performance  modeling  has  been  used  to  provide  estimates  of 
training  time  and  performance  levels  with  sample  data  (Schneider, 
1990) .  One  innovative  approach  which  applies  to  modeling  was 
developed  by  Kennedy  and  his  colleagues  (Kennedy,  Jones,  & 
Baltzley,  1988a,  1988b) .  Kennedy  et  al.  developed  a  model  known 
as  the  isoperformance  model.  This  model  makes  assumptions  about 
the  relationship  between  training  variables,  personnel  variables 
(e.g.,  aptitude),  and  system  performance  and  divines  an  optimum 
performance  level  (thus  the  word  'isoperformance' )  among  these 
variables . 

Isoperformance  is  based  on  system  "trade-offs";  that  is,  how 
a  level  of  operational  performance  (or  minimum  cost)  can  be 
obtained  through  different  combinations  of  personnel  (i.e., 
individual  differences),  training  (e.g.,  number  and  length  of 
trials),  and  equipment  (e.g.,  varied  features  on  a  single  machine 
or  different  engineering  options) .  Isoperformance  also  refers  to 
a  curve  defining  the  relationship  of  training  time  and  individual 
ability.  In  other  words,  an  isoperformance  curve  shows  the 
amount  of  training  time  needed  by  persons  of  differing  ability  to 
beoome  proficient  on  a  system.  For  example,  as  seen  in  Figure  3, 
persons  with  higher  aptitude  require  less  training  time  to  become 
80%,  50%,  or  20%  proficient  on  a  system  than  persons  with  lower 
aptitude.  Although  most  isoperformance  curves  are  negatively 
accelerated,  they  can  have  other  shapes  depending  upon  the 
distribution  of  individual  aptitudes.  For  example,  in  the 
military,  many  job  specialties  require  the  exclusion  of  low 
aptitude  personnel.  This  causes  the  isoperformance  curve  to  drop 
sharply  at  first  and  then  descend  in  a  roughly  linear  manner 
(Kennedy  et  al.,  1988b). 

According  to  Kennedy  et  al .  (1988a,  1968b),  isoperformance 
can  be  used  in  a  variety  of  ways.  For  example,  isoperformance 
curves  can  be  used  to  assess  what  aptitude  category  is  needed  to 
fill  certain  positions  in  the  military.  Isoperf or.mance  can  also 
be  used  to  evaluate  current  systems  (i.e..  to  suggest 
improvements  or  to  compare  the  costs  of  different  parts  of  the 
system) . 

The  isoperformance  model  appears  to  be  an  application  of 
learning  curves  to  trade-offs  among  known  system  variables  (i.e., 
training,  individual  differences,  and  sy.<item  performance)  .  It 
appears  to  add  nothing  new  to  the  development  of  valid  learning 
curves  themselves.  It  does  appear  to  serve  well  in  the  context 
of  a  series  of  conceptual  tools  on  which  to  apply  empirically- 
based  learning  curves,  once  they  are  available.  Presently,  this 
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Figure  3.  Isoperformance  curves  for  80/  50,  and  20%  proficiency 
(adapted  from  Kennedy  et  al./  1988). 


line  of  research  suffers  from  a  distinct  lack  of  systematic  data. 

Issues  Relevant  to  the  Measurement  and  Prediction  of  the  Effects 

of  Training  on  Performance 

A  number  of  related  topics  arise  when  discussing  the 
measurement  and  prediction  of  the  effects  of  training  variables 
on  the  performanv^e  of  military  tasks.  Some  of  the  more  important 
include : 


•  Transfer  of  Training  (TOT) 

•  Training  Effectiveness  Evaluation 

•  Performance  Measurement 

•  Experimental  and  Quasi-experimental  Designs 

•  Simulator  Fidelity 

•  Task  and  Behavioral  Taxonomies 

While  each  of  these  bear  important  tangential  relations  to 
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the  topic,  it  is  beyond  the  scope  of  this  report  to  describe  in 
detail  the  importance  of  each  to  the  measurement  and  prediction 
of  human  reliability  growth  as  defined  in  this  report.  Instead, 
a  summary  of  each  topic,  including  a  definition  of  each  topic  and 
its  general  ramifications  to  the  problem  domain,  is  provided.  A 
topically  arranged  bibliography  of  the  major  research  in  each 
area  is  enclosed  as  Appendix  E. 

Transfer  of  Training  (TOT) 

TOT,  or  more  generally  transfer  of  learning,  occurs  when  the 
learning  of  one  task(s)  generalizes  to  other,  similar^  tasks. 
According  to  Valverde  (1973),  "Anything  which  the  trainee  can 
learn  can  be  transferred,  including  skills,  facts,  learning  sets, 
self-confidence,  interests,  and  attitudes.  Transfer  may  be 
either  positive  or  negative.  Positive  transfer  takes  place  when 
the  acquisition  of  one  skill  facilitates  the  acquisition  of 
another  skill.  Negative  transfer  occurs  when  the  acquisition  of 
one  skill  interferes  with  the  acquisition  of  another  skill." 

Derived  measures  of  TOT  are  numerous,  including: 

•  Percent  of  transfer  (Gagne,  Foster,  &  Crowley, 

1948) 

•  Incremental  transfer  effectiveness  (Roscoe,  1971) 

•  Transfer  effectiveness  ratio  (Roscoe,  1972) 

Other  criterion-referenced  performance  measures  often  used 
in  TOT  studies  are  listed  under  the  performance  measurement  topic 
below. 

Lane  (1986b)  notes  that  the  ultimate  criterion  of  training 
effectiveness  is  the  capability  to  perform  a  job.  Transfer  is 
(along  with  retention  of  learning)  a  major  intermediate  criterion 
of  successful  training.  The  relationship  between  acquisition  of 
skilled  performance  and  the  application  (meaning:  successful 
transfer  of  that  skilled  performance)  are  not  "straightforward, " 
in  Lane's  own  words.  Some  training  methods  have  positive  effects 
on  transfer  simply  because  they  allow  the  emergence  of  higher 
levels  of  training  performance  and  learning.  However,  there  are 
trade-offs.  Over-training  (over-learning)  can  reduce  transfer  in 
some  situations.  This  may  be  due  to  the  automaticity  that  occurs 
with  overlearning  skills  through  too  much  practice.  Students  may 
lack  the  flexibility  to  see  subtle  cue  or  context  differences 
among  otherwise  similar  tasks,  that  may  require  differences  in 
their  responses.  Witness  the  accident  at  Three  Mile  Island 
(Rogovin  &  Frampton,  1979),  in  which  control  room  operators 


’it  is  the  word,  similar,  that  draws  the  largest  debate  among  military  training 
researchers.  Theories  for  'how  similar'  (meaning  how  generallzable)  exist  without  general 
consensus  (e.g.,  Osgood's  1949  transfer  surface  paradigm),  but  empirical  evidence  for 
similarity  (read:  generalizability)  which  supports  any  theory  to  date  is  lacking. 
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followed  their  training  perfectly  -  for  the  wrong  accident 
situation.  They  failed  to  properly  interpret  the  cues  that  would 
allow  them  to  properly  diagnose  the  true  plant  status.  Correct 
diagnosis  would  then  have  led  them  to  perform  the  correct 
procedures  for  plant  management.  Other  examples  exist,  but  it 
should  suffice  that  the  relationship  between  TOT  and  the 
relationship  of  training  variables  and  performance  is  not  well 
defined. 

Training  Effectiveness  Evaluation  (TEE) 

TEE  is  a  methodology  for  evaluating  the  effectiveness  of 
training,  primarily  training  devices  and  simulators.  According 
to  Jeantheau  (1971) ,  the  following  techniques  may  be  applied  in 
the  context  of  a  TEE: 

•  Qualitative  Assessment 

•  Non-comparative  Measurement 

•  Training  Device/Conventional  Comparison 

•  Utilization  Procedure  Comparison 

•  Training  Device  Comparison 

•  TOT 

Other  researchers  also  consider  cost  effectiveness  comparisons  as 
a  method  of  TEE. 

Only  the  latter  two  in  the  above  list.  Training  Device 
Comparison  and  TOT  (covered  above),  are  immediately  relevant  to 
the  question  of  the  relationship  between  training  variables  and 
performance.  Usually  the  training  variables  evaluated  are  device 
characteristics  and  utilization  procedures.  Reports  which 
present  results  of  rigorous  TEEs  may  be  usable  in  defining  the 
data  base  for  a  HRGM. 

Performance  Measurement 

Performance  measurement  is  the  collection  of  human 
performance  data  in  either  the  learning/training  environment  or 
on  the  job.  It  may  be  subjective  or  objective,  as  well  as 
instrumented  (using  a  computer  set  up  to  record  performance 
measures  automatically) .  The  relationship  of  performance 
measurement  to  human  reliability  growth  is  a  potentially  negative 
one.  Invalid,  inaccurate  performance  measures  can  negate  the 
value  of  the  growth  model's  predictions.  It  is  that  simple. 
Unfortunately,  it  is  not  that  simple  to  control  the  precision  of 
the  performance  measures. 

Lane  (1986a)  identifies  seven  criteria  for  evaluating 
performance  measures  that  include; 

(a)  Reliability 

(b)  Validity 
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(c)  Sensitivity 

(d)  Completeness 

(e)  Separability  of  operator  from  contributions  of  the 
measurement  context 

(f)  Diagnosticity  (specificity) 

(g)  Utility  and  cost  benefit  (value  against  alternatives) . 

Of  these  criteria,  it  appears  that  a.,  b.,  c.,  and  f.  would  have 
the  most  significant  implications  for  the  growth  model. 

Experimental  and  Ouasi-experimental  Design 

Most  TOT  studies,  TEEs,  and  applications  of  performance 
measurement  to  the  training  environment  make  use  of  experimental 
design  and  statistics.  This  striving  for  experimental  rigor  has 
also  placed  a  significant  expense  on  the  ability  to  collect 
adequate  training  and  performance  data.  While  quasi-experimental 
designs  have  relieved  some  of  the  burden  on  the  experimenter  to 
collect  a  large  enough  data  set  from  a  large  enough  sample  size, 
they  have  done  so  at  the  expense  of  permitting  confounds  to  exist 
among  important  training  variables.  Recent  efforts  to  optimize 
experimental  rigor  and  to  make  more  economical  use  of  data 
collection  and  subject  pools  include  response  surface  methodology 
(Simon,  1979)  and  fitting  learning  curves  using  orthogonal 
polynomials  (Sabol,  1986).  Sabol's  use  of  orthogonal  polynomials 
is  of  particular  interest  because  it  attempts  to  make  use  of  the 
learning  curve  tendency  that  occurs  in  formal  training  to  analyze 
specific  components  that  can  be  tested  for  statistical 
significance.  Unfortunately,  the  technique  does  not  provide  the 
oppcrtunity  for  direct  determination  of  the  form  of  the  learning 
curve  equation. 

Simulator  Fidelity 

Fidelity  in  training  to  the  operational  environment  is  a 
major  focus  in  the  investment  of  training  managers  for  training. 
Unfortunately,  the  rules  for  fidelity  levels  required  are  as 
unknown  as  the  degree  of  similarity  among  tasks  required  for 
transfer  of  training  to  occur.  Fidelity  is  one  of  the  important 
training  variables  that  a  HRGM  would  hope  to  predict .  Current 
knowledge  in  this  area  does  not  support  a  successful  outcome  with 
respect  to  this  goal. 

Task  and  Behavioral  Taxonomies 

The  use  of  task  or  behavior  classification  schemes  is  an 
intrinsic  part  of  the  construction  and  operation  of  a  HRGM.  It 
represents  the  primary  means  by  which  the  learning  and 
performance  data  and  relationships  could  be  accessed  for  use  in 
making  projections.  A  comprehensive  review  of  existing  task  and 
behavior  taxonomies  was  beyond  the  scope  of  this  project. 
Certainly,  the  development  of  a  completely  new  and  unique 
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taxonomy  was  deemed  highly  undesirable  and  was  quickly  dropped 
from  further  consideration,  particularly  in  light  of  the 
uncertainties  which  exist  with  regard  to  the  ability  to  collect 
sufficient  quantities  of  adequate  and  usable  training  and 
performance  data  on  which  to  develop  a  growth  model .  The 
following  three  references  were  reviewed  for  application  to  the 
taxonomic  concerns  of  this  project: 

O'Brien,  Simon,  &  Swaminathan  (1989)  Development  of  PER- 
SEVAL  Performance  Shaping  Functions. 

Hritz  (1979)  Behavior  Taxonomies  and  Training  Equipment 
Design:  A  Literature  Review  and  General  Model. 

Bownas  &  Cooper  (1978)  Review  of  Reported  Behavior  and 
Performance  Taxonomies. 

The  taxonomy  chosen  was  contained  in  the  O'Brien  et  al.  report, 
as  will  be  discussed  in  a  later  section  of  this  chapter. 


Summary  of  the  Findings  Based  on  the  Review  of  the 
Theoretical  Literature 

The  review  of  the  literature  presented  above  provided  the 
basis  for  the  following  four  objectives: 

1.  Identify  the  metrics  and  methodology  for  use  in  the 
development  of  a  HRGM  to  predict  the  effect  of  training 
variables  on  soldier  performance. 

2.  Identify  the  tasks  (i.e.,  taxa)  on  which  to  obtain  and 
classify  training  and  performance  data  required  to 
support  Objective  #1  above. 

3.  Identify  the  kinds  of  data  required  to  support 
Objective  #1  above  and  potential  sources  of  such  data. 

4.  Develop  the  criteria  for  determining  the  acceptability 
of  actuarial  data  for  use  in  constructing  a  proposed 
HRGM . 

Each  of  these  objectives  is  discussed  in  the  remainder  of  this 
chapter . 

Metrics  and  Methodology  for  Use  in  Developing  a  HRGM 

The  metrics  deemed  appropriate  for  use  in  the  development  of 
a  HRGM  that  predict  the  relationship  of  training  variables  to 
soldier  performance  are,  not  surprisingly,  the  typical  learning 
curves  already  reviewed.  In  general,  three  of  the  four  major 
types  of  learning  curve  equation  forms  are  considered  most  useful 
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as  metrics  to  employ  in  such  a  model.  These  are:  (a)  the  power 
curve,  (b)  the  hyperbolic,  and  (c)  the  exponential.  The  logistic 
function  is  the  one  form  of  the  learning  curve  equation  that  may 
not  be  appropriate.  This  is  due  to  our  agreement  with  the 
literature  (Lane,  1986b)  that  the  logistic  function  may  occur 
only  when  the  source  of  data  is  subjective  input  from  subject 
matter  experts  (SMEs)  such  as  instructors.  The  logistic  function 
may  map  only  the  SMEs  interpretation  of  student  learning 
acquisition  rates.  As  a  consequence,  we  will  not  consider  the 
logistic  function  as  a  metric  to  be  considered  as  part  of  the 
growth  model . 

Among  the  four  parameters.  Identified  by  Spears  (1983)  as 
important  aspects  of  learning  curves  to  consider  in  evaluating 
various  aspects  of  learning  and  performance,  two  parameters 
appear  to  dominate.  These  include  the  rate  of  learning  and 
asymptote  parameters.  Data  which  measure  these  parameters  would 
be  most  valuable  in  evaluating  the  effects  of  training  variables 
on  soldier  performance.  The  other  two  parameters  espoused  by 
Spears  (1983),  beginning  learning  and  inflection  point,  are 
considered  less  valuable  for  two  reasons.  First,  the  literature 
does  not  reflect  any  )cnown  applications  of  these  two  parameters. 
Second,  their  range  of  potential  application  to  problems  of 
predicting  the  effects  of  training  variables  on  soldier 
performance  is  narrow,  which  malces  them  of  limited  value  to  a 
growth  model. 

The  types  of  measures  used  in  the  literature  for  the 
measurement  of  both  training  (independent  variables)  and 
performance  (dependent)  are  as  shown  in  Table  1.  These  measures 
appear  the  most  likely  and  more  suitable  to  obtain  for  use  in 
constructing  a  HRGM. 

The  considerations  discussed  above  provided  the  basis  for 
collecting  literature  which  reports  actual  findings  at  a  level  of 
resolution  usable  for  constructing  a  model.  Once  obtained,  the 
data  could  then  be  reviewed  and  those  that  fit  the  project 
requirement  selected.  This  subset  of  data  would  then  be 
analyzed,  and  a  "best  fit"  learning  curve  could  be  developed.  A 
"best-fit"  learning  curve  is  one  that  accounts  for  the  most 
variance  in  the  data  or  the  highest  r*  value  (r  is  the 
correlation  coefficient) .  One  way  to  examine  the  fit  of  a 
learning  curve  to  the  data  would  be  to  transform  the  dependent 
and  independent  variables  logarithmically  into  log-log 
coordinates  using  the  power  function.  It  is  also  possible  to 
transform  only  one  of  the  variables  logarithmically  into  a  semi- 
logarithmic  or  semi-log  coordinates  (the  exponential  function) 
(Lane,  1986b) . 
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Table  1 

Measures  Used  for  Both  Training  and  Performance 

A.  Measures  Used  to  Assess  Training  Performance. 

•  Time  to  learn 

•  Rate  of  learning 

•  Errors  committed 

•  Performance  time  on  trials/practice 

•  Asymptote  level  of  learning 

•  Training  teat  scores  (e.g.,  on  job 
knowledge  tests) 

•  Instructor  ratings 

•  Simulator  performance  data  (instrumented 
data) 

•  'Schoolhouse'  grades 

B.  Measures  Used  to  Assess  Job  Performance 

•  Time  to  perform  (and  task-based 
variations) 

•  Accuracy  (and  task-based  variations) 

•  Instrumented  performance  data 


The  curves  derived  from  the  data  could  then  be  used  in  a 
growth  model  to  form  specific  trade-off  analyses  and/or  "what-if” 
type  analyses  based  on  the  relationship  between  training  and 
performance  depicted  by  the  curves.  The  curves  could  also  be 
used  to  make  specific  predictions  concerning  a  particular  task 
type  (or  taxa) .  For  example,  if  data  were  gathered  from  the 
training  of  perceptual-motor  tasks  in  addition  to  cognitive  skill 
tasks  and  motor  tasks,  then  the  perceptual-motor  data  would  not 
only  be  useful  in  making  general  predictions  about  training  and 
performance  but  would  also  be  useful  in  specific  predictions 
concerning  training  and  performance  with  perceptual-motor  tasks. 

Identification  of  Taxa  for  Inclusion  in  the  Library  of  Learning 
Curves  Used  in  the  HRGM 

A  HRGM  requires  the  use  of  a  series  of  taxa  (meaning 
globally  classified  behavioral  tasks  with  strong  military 
relevance)  in  order  to  organize  and  access  the  learning  curve 
functions  which  might  be  available.  This  would  facilitate  the 
use  of  the  growth  model  in  predicting  the  effect  of  training 
variables  on  performance  by  providing  ease  of  access  to  the 
empirically-based  learning  curve  algorithms  contained  in  the 
model.  While  the  development  of  a  formal  taxonomy  of 
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behavlorally  anchored  taxa  was  deemed  as  outside  of  the  scope  of 
this  effort,  especially  since  several  task  and  behavioral 
taxonomies  are  already  in  existence  (e.g.,  Berliner,  Angell,  & 
Shearer,  1964),  there  was  a  need  to  provide  a  task-based 
structure  to  the  collection  of  relevant  learning  curve  data. 
Several  behavioral  taxonomic  models  were  reviewed  (O'Brien  et . 
al.,  1989;  Hritz  1979/  Bownas  &  Cooper,  1978)  and  compared  for 
utility  to  this  project.  It  was  decided  to  select  the  basic 
taxonomic  listing  used  by  O'Brien  et  al.,  since  this  selection 
would  make  it  feasible  to  extend  the  use  of  the  data  collected  in 
this  project  for  application  to  the  ongoing  ARI  research  program 
to  develop  a  family  of  computer  simulation  models.  These  data 
would  have  greatest  bearing  on  the  PER-SEVAL  model  member  of  this 
family  of  computer  models.  The  taxa  contained  in  the  O'Brien  et 
al .  research  are  shown  in  Table  2 . 

Table  2 

PER-SEVAL  Task  Taxonomy 


Type  Taxon 

Perceptual  Visual  Recognition/ 

Discrimination 


Cognitive  Numerical  Analysis 

Information  Processing/ 
Problem  Solving 


Motor  Fine  Motor-Discrete 

Fine  Motor  Continuous 
Gross  Motor  -  Light 
Gross  Motor  ~  Heavy 


Communication  Oral 

Reading  and  Writing 


O'Brien  et  al .  (1989)  state;  "The  PER-SEVAL  task  taxonomy  is 
primarily  an  expansion  of  Berliner's  (1964)  task  taxonomy." 
Several  of  the  Berliner'  taxa  were  eliminated  from  the  final  list 


employed  by  O'Brien  et  al.  on  the  basis  of  a  lack  of  military 
relevance  (i.e.,  tasks  which  occur  either  not  at  all  or 
infrequently  in  the  military  context) . 

The  primary  application  of  the  PER-SEVAL  task  taxonomy  was 
to  guide  the  literature  search  and  review  activities  which  made 
up  the  predominate  effort  in  this  project.  It  also  provided  the 
basis  for  classifying  data.  All  subsequent  descriptions  or 
listings  of  the  data  described  in  this  report  are  in  accordance 
with  this  basic  task  taxonomy. 

Data  Recruirements  and  Sources  of  Training  and  Performance  Data 

In  order  to  apply  curve  fitting  techniques  to  assess  the 
relation  of  training  and  performance,  relevant  data  were  required 
from  a  variety  of  sources.  These  sources  could  potentially 
include ; 

•  Military  and  non-military  social  and  behavioral  science 
databases  (e.g.,  PsycINFO,  ERIC,  DTIC,  NTIS,  NASA), 

•  Data  from  other  sources  such  as  personal  libraries  and 
unpublished  reports, 

•  Computerized  data  bases  of  training  and  performance 
data  maintained  by  the  major  DOD  behavioral  and  social 
science  laboratories  (TPDC,  AFHRL,  NPRDC,  ARI) ,  and 

•  Collection  of  new  training  and  performance  data  for  use 
in  developing  new  learning  curve  relations. 

Of  the  four  types  of  data  sources,  this  project  was  limited  to 
the  first  three.  Collection  of  new  data  was  beyond  the  scope  of 
work  of  this  project.  The  search  techniques  used  in  obtaining 
these  data  are  discussed  in  the  next  chapter.  The  issue  of  data 
collection  to  support  this  line  of  research  is  covered  in  Chapter 
4.  Chapter  4  also  covers  conclusions  and  recommendations. 

Criteria  for  the  Acceptability  of  Data  Used  in  the  Development  of 
a  HRGM 

The  criteria  to  be  applied  to  the  data  to  be  considered  for 
use  in  the  development  of  a  HRGM  are  presented  in  Table  3.  These 
criteria,  derived  from  our  understanding  of  the  underlying 
research  literature  on  human  learning  and  practice,  the  use  and 
application  of  learning  curve  fitting  techniques  to  the  military 
training  context,  and  the  nature  and  sources  of  potential 
training  .*»nd  job  performance  data,  are  intended  to  assist  us  in 
answering  the  basic  question: 

Do  sufficient  quantities  of  suitable  data  exist  on  which  to 

develop  a  credible  HRGM? 
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An  affirmative  answer  to  this  question  then  leads  to  the 
conceptualization  of  the  model  parameters  and  constructs  which 
would  be  Incorporated  Into  such  a  model.  A  negative  answer  to 
this  question,  however,  leads  to  the  conclusion  that  without  such 
suitable  data,  such  a  stand  alone  model  cannot  be  developed  based 
on  the  available  empirical  data.  This  question  and  Its  outcome 
form  the  basis  for  the  remainder  of  this  report. 

Table  3 

Criteria  for  Evaluating  the  Adequacy  of  the  Data  for 

Use  in  Developing  an  HRGM 


Existence  of  actuarial  data,  which  meet 
the  following  subcriteria: 

• 

Measurement  of  both  predictor 
(measures  of  training)  and  the 
criterion  (job  performance) 

• 

Measurement  that  preserves  the 
accounting  for  individual 
differences  or  of  meaningful 
groups  (bloc}cs)  of  subjects 
(e.g.,  AFQT  category) 

Inter-study  comparability  of 
independent  and  dependent 
variables  within  taxa 

• 

Same  independent  and  dependent 
measures  (or  transformable  to 
same  independent  and  dependent 
measures)  employed  across 
studies 

• 

Sample  size  of  study  from  which 
data  are  derived 

Relevance 

to  military  task  requirements 

Relevance  to  military  training 
environment 

Comprehensiveness  of  Taxa  coverage 

Itethodology  for  tho  Collootlon 
and  Raduotlon  of  Quantltativa  Human  Parfoxmanca  Data 
Ralavant  to  Taak  Training 


The  collection  and  reduction  of  quantitative  human 
performance  data  was  required  in  order  to  answer  the  question 
previously  posed  in  the  last  chapter,  that  is; 

Do  sufficient  quantities  of  suitable  data  exist  on  which  to 
develop  a  credible  HRGM? 

The  initial  effort  to  answer  this  question  involved  a  meta- 
analytic  model.  As  will  be  explained,  this  approach  had  to  be 
abandoned  and  a  less  rigorous  approach  was  devised  in  which 
suitable  data  were  collected  in  whatever  form  they  were  presented 
and  classified  within  the  PER-SEVAL  taxonomy  (O'Brien  et  al., 

1989) .  The  discussion  that  follows  presents  both  of  these 
approaches  as  well  as  a  detailed  description  of  the  literature 
review  and  results  obtained. 

Initial  Approach;  Meta-Analvsis 

The  first  step  in  developing  a  HRGM  was  to  derive 
quantitative  relationships  of  the  impact  of  training  on 
performance  from  the  human  behavioral  and  social  science 
literature  using  meta-analysis .  Meta-analysis  is  a  statistical 
procedure  for  combining  research  results  of  a  'similar  nature  and 
kind'  across  numerous  studies.^  For  example,  meta-analysis  has 


'  Meta-analysis  begins  with  the  formulation  of  questions  or  hypotheses  concerning 
some  variable  of  Interest.  In  our  case,  the  variables  of  Interest  would  be  training 
variables,  such  as: 

•  different  methods  of  instruction, 

•  different  equipment,  part  task  versus  whole  task  training  devices,  and 

•  training  time. 

The  key  to  this  effort  Is  the  effect  of  these  training  variables  on  human  performance. 
For  example,  one  approach  would  be  to  study  the  effect  of  part  task  versus  whole  task 
training  on  the  learning  of  a  visual  discrimination/information  processing  task  such  as 
pursuit  tracking.  Next,  articles  relevant  to  the  research  question  would  be  collected 
from  various  sources  Including  literature  searches,  journals,  and  private  libraries,  and 
screened  for  Inclusion  Into  the  meta-analysls.  The  screening  process  would  ascertain: 
<1)  whether  the  articles  were  relevant  to  the  research  question,  and  (2)  whether  the 
articles  contained  the  statistical  Information  needed  to  conduct  the  meta-analysls. 

The  results  from  each  of  the  articles  have  to  be  converted  into  a  common  statistic 
under  the  rules  of  meta-analysls.  As  a  result,  each  article  should  report  the  results 
of  statistical  tests  such  as  the  descriptive  statistics  of  £,  x’,  r',  or,  at  a  minimum, 
the  mean,  standard  deviation,  and  sample  size.  The  next ^tep  would  be  to  convert  the 
results  from,  each  article  Into  a  common  descriptive  statistic.  One  of  the  most  widely 
Used  descriptive  statistics  In  meta-analysls  Is  the  Index  of  effect  size  or  the  "d" 
statistic  (Cohen,  1977) .  The  d  statistic  is  a  transform  of  the  polnt-blserlal  correlation 
(Hunter  t  Schmidt,  1990),  Specifically,  It  Is  the  difference  between  the  means  of  the 
control  and  experimental  groups  divided  by  the  within  group  variance  (l.e.,  the  sum  of 
the  variances  of  the  experimental  and  control  groups  divided  by  2)  .  However,  other 
statistics  may  be  used  In  meta-analysls  such  as  the  polnt-blserlal  correlation  r  and  the 
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been  successfully  applied  to  determine  the  effectiveness  of 
psychotherapy  (Glass,  1977)  and  to  examine  the  relationship 
between  spatial  experience  and  spatial  test  performance 
(Baennlnger  &  Newcombe,  1989) .  The  results  of  each  meta-analysls 
would  be  organized  by  a  predefined  set  of  globally  classified 
tasks  or  taxa  with  a  strong  military  relevance  (l.e.,  the  PER- 
SEVAL  taxonomy  by  O'^Brlen  et  al.,  1989). 

However,  our  Initial  approach  soon  wavered  and  was 
subsequently  revised  because  several  problems  emerged  which 
affected  the  feasibility  of  using  meta-analysls  to  develop  the 
model.  A  major  problem  was  the  tremendous  diversity  of  task 
types  (e.g.,  some  tasks  were  composed  of  single  component 
operations  while  others  were  multi-component)  found  in  the  data 
within  each  of  the  taxa.  In  addition,  the  types  of  training  used 
within  and  between  the  taxa  were  as  diverse  as  the  task  types 
were  within.  Both  of  these  problems  made  logical  groupings  of 
related  experiments  virtually  impossible  to  derive.  Another 
problem  was  the  uneven  distribution  of  studies  for  the  taxa; 
there  were  no  studies  representing  the  taxa  of  numerical 
analysis,  gross  motor-light,  and  communication-oral  while  the 
taxa  of  visual  information  processing  and  information  processing 
were  relatively  data-rich.  There  was  also  a  problem  in  how  to 
Incorporate  multiple  Independent  variables  within  one  research 
article  into  the  meta-analysls.  Thus,  it  was  concluded  that  a 
meta-analysls  of  the  Impact  of  training  variables  on  performance 
organized  by  behavioral  taxa  could  not  be  supported  by  the  extant 
data. 


Our  problems  in  obtaining  suitable  data  to  support  the  meta- 
analysls  are  not  altogether  surprising  because  similar  problems 
have  been  reported  elsewhere  by  Kennedy  et  al.  (1988a/  1988b). 
Specifically,  Kennedy  et  al.  surveyed  the  systems  research  and 
human  factors  engineering  literature  for  studies  comparing 
equipment,  aptitude,  and  training.  Of  approximately  10,000 
articles  searched,  only  276  involved  studies  of  training  and 


the  variances  of  the  experimental  and  control  groups  divided  by  2)  .  However,  other 
statistics  may  be  used  In  meta-analysls  such  as  the  pclnt-blserlal  correlation  r  and  the 
population  strength  between  the  Independent  and  dependent  variables  or  omega  squared 
(Hays,  1973). 

The  next  step  In  the  meta-analysls  would  be  to  test  the  effect  sizes  of  the  studies 
for  homogeneity  of  variance.  If  the  variances  were  homogeneous,  then  a  combined  effect 
size  would  be  computed.  The  combined  effect  size  Is  a  sum  of  the  effect  sizes  divided 
by  the  number  of  studies.  Finally,  the  null  hypothesis  would  be  accepted  or  rejected 
depending  on  whether  or  not  It  fell  between  prescribed  confidence  Intervals  (for  a  more 
detailed  description  of  meta-analysls.  see  Glass.  1977;  Hunter  &  Schmidt.  1990) . 

The  results  from  the  meta-analysls  would  then  be  used  to  develop  a  model  or 
algorithms  to  predict  the  impact  of  training  variables  on  performance.  Predicting  the 
effect  of  training  variables  on  performance  would  be  accomplished  by  either  employing 
curve  fitting  techniques  to  derive  taxa-based  learning  curves  or  by  building  regression 
models. 
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performance  as  a  function  of  equipment  variations.  From  this, 
only  10  articles  reported  information  necessary  to  calculate  the 
combined  inferential  statistic,  omega-squared.  Although  there 
were  no  problems  in  calculating  omega-squared  in  each  study, 
Kennedy  et  al.  (1988b)  reported  that  the  data  "turned  out  to  be 
too  irregular  to  permit  sufficient  generalizations  about  trends 
in  these  studies"  (p.  59) .  In  a  similar  vein.  Lane  (1986b) 
reports  that  published  research  on  acquisition  and  learning  could 
not  support  generalized  predictions  concerning  the  time  course  of 
training  across  different  training  situations  with  learning 
curves . 

Revised  Approach!  Generalized  Classification  of  Learning 
and  Performance  Literature 

Our  revised  approach,  therefore,  involved  locating  suitable 
research  articles  which  presented  data  and  or  learning  curves 
presenting  the  relation  of  training  to  performance.  Essentially, 
the  technical  approach  involved  collecting  and  reviewing  data 
from  the  literature  and,  if  feasible,  analyzing  the  data  to 
derive  learning  curves.  The  curves  derived  from  the  data  would 
then  be  usable  in  a  growth  model  to  perform  specific  "trade-off" 
and  "what-if"  analyses  based  on  the  relationship  between  training 
and  performance  depicted  in  the  learning  curves.  No  special 
assumptions  were  made  about  the  data  obtained  at  this  point  in 
order  to  permit  the  widest  possible  'net'  in  our  data  capturing 
effort.  The  discussion  that  follows  presents  the  approach  used 
and  results  obtained  during  the  literature  search. 

Initial  Literature  Search 

The  difficulty  in  identifying  suitable  sources  of  human 
performance  data  related  to  training  is  underscored  by  the 
difficulties  encountered  in  performing  the  literature  search. 

Two  efforts  were  made  to  use  automated  literature  search 
techniques.  The  initial  literature  search  was  structured  to 
obtain  a  list  of  publications  relevant  to  TRAINING  and  HUMAN 
PERFORMANCE  in  accordance  with  each  of  the  taxa  contained  in  the 
PER-SEVAL  taxonomy.  Table  4  summarizes  this  search  strategy. 

The  literature  search  commenced  with  a  review  of  the 
following  on-line  databases  using  this  literature  search 
strategy: 


•  Defense  Technical  Information  Center  (DTIC) , 

•  National  Technical  Information  Service  (NTIS) , 

•  Educational  Resources  Information  Center  (ERIC) , 

•  Psychological  Abstracts  Information  Services 
(PsycINFO) ,  and 

•  SOCIAL  SCISEARCH. 
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Table  4 

Keywords  Used  in  the  Initial  Literature  Search 


"Training”  and  "Performance"  and  each 
of  the  following  individually: 

Visual  Recognition/Discrimination 
Numerical  Analysis 
Information  Processing 
Fine  Motor  (discrete) 

Fine  Motor  (continuous) 

Gross  Motor  (heavy) 

Gross  Motor  Uight) 
Communication  (read  and  write) 
Communication  (oral) 


Not  only  were  the  selected  taxa  used  as  keywords,  but  they 
also  became  the  principal  means  of  classifying  the  relevant 
literature  identified  through  the  literature  search, 

A  print-out  of  titles  (over  3,000  separate  titles  in  our 
estimate)*  of  literature  citations  derived  from  the  search  was 
obtained  and  approximately  500  citations  were  identified  as  being 
candidate  publications.  Among  the  citations  were  articles  from 
the  broad  behavioral  science  and  human  factors  literature 
relevant  to  training  and  human  performance  and  at  least  one  of 
the  nine  taxa  listed  above.  A  further  request  was  made  to  obtain 
a  print  out  of  the  abstracts  of  the  500  candidate  articles. 

These  abstracts  were  then  carefully  evaluated  in  terms  of: 

•  apparent  probability  of  the  presence  of  quantitative 
data, 

•  appropriate  statistical  design  and  analysis, 

•  proper  population  (i.e.,  normal  adult  population),  and 

•  (to  a  lesser  extent)  content. 

This  process  resulted  in  the  selection  of  108  acceptable 
abstracts. 

The  next  step  involved  the  collection  of  publications. 
Attempts  were  made  to  obtain  the  108  candidate  publications  from 


■  ^  ~ 

*  It  was  difficult  to  determine  the  number  of  original  "hits"  in  the  literature, 
since  there  were  so  many  repetitions  of  titles  across  search  terms  and  data  bases. 
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several  sources,  including  the  ARI  Headquarters  Technical  Library 
and  several  local  university  libraries .  The  candidate 
publications  were  then  evaluated  in  terms  of  appropriateness  of 
the  data,  statistics,  and  content.  This ' procedure  resulted  in 
the  selection  of  42  articles.  Moreover,  this  stage  marked  the 
establishment  of  the  visual  and  information  processing  taxa  as 
the  most  data-rich  in  terms  of  available  relevant  literature 
(i.e.,  all  but  6  of  the  initial  42  studies  examined  were 
classified  as  either  visual  recognition/discrimination  or 
information  processing) . 

Expansion  of  the  Literature  Base 

These  initial  42  articles  were  summarized  and  presented  in 
an  In-Process  Review  to  ARI  scientists.  Upon  further 
examination,  it  was  concluded  that  only  5  of  the  42  articles 
contained  predictor  and  criterion  variables  appropriate  to  the 
project.  After  discussion  with  ARI  scientists,  it  was  decided 
that  an  expansion  of  the  literature  base  was  a  necessary  next 
step  in  order  to  assure  ourselves  that  the  search  strategy  was 
not  the  cause  of  the  failure  to  find  larger  amounts  of  suitable 
data.  Two  initiatives  commence  to  expand  the  literature  base: 

•  search  of  other  government  laboratory  data  bases,  and 

•  development  and  implementation  of  new  keywords  in 
future  searches . 

Search  of  government  data  bases.  Major  government 
laboratories  were  contacted  to  determine  the  existence  and 
availability  of  relevant  data,  including: 

•  Training  and  Performance  Data  Center  (TPDC) , 

•  Naval  Personnel  Research  and  Development  Center 
(NPRDC) , 

•  Air  Force  Human  Resources  Laboratory  (AFHRL) ,  and 

•  Army  Research  Institute/Manpower  and  Personnel  Research 
Laboratory  (ARI/MPRL) . 

Summaries  of  the  discussions  with  each  organization  are  presented 
below. 

Points-of  contact  at  TPDC  expressed  doubt  as  to  the 
existence  of  training  and  performance  data  relevant  to  the 
project.  It  was  suggested,  however,  that  we  contact  ARI/MRPL 
representatives  for  possible  data.  Subsequent  discussion  with 
MPRL  personnel  informed  us  that  only  attitudinal  data  were 
presently  being  collected  at  ARI  (which  would  not  be  relevant  to 
this  project) . 
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Representatives  of  NPRDC  were  also  contacted  with  respect  to 
the  availability  of  training  and  performance  data.  It  was 
revealed  that  NPRDC  had  an  extensive  training  data  base,  but  job 
performance  data  were  not  collected.  Additional  conversation 
with  NPRDC  representatives  in  San  Diego  disclosed  that  training 
and  performance  data  had  been  collected,  but  several 
methodological  confounds  precluded  the  usefulness  of  the  data  for 
our  purposes. 

Continued  efforts  were  made  to  obtain  information  pertaining 
to  the  availability  of  training  and  performance  data  from 
government  laboratories.  Contact  with  AFHRL  in  San  Antonio 
revealed  that  the  Air  Force  has  ongoing  training  projects  with 
emphasis  on  planned  instruction,  but  no  appropriate  data  have 
been  collected. 

Implementation  of  new  keywords.  The  second  method  by  which 
attempts  were  made  to  expand  the  literature  base  involved  the 
implementation  of  new  keywords  derived  from  the  5  articles 
containing  predictor  and  criterion  variables.  The  new  keywords 
were  used  to  peruse  areas  of  the  literature  that  were  unexplored 
in  the  initial  search.  These  additional  keywords  are  shown  in 
Table  5.  As  a  preliminary  examination,  these  keywords  were  used 
in  an  interactive  search  of  the  on-line  data  bases  of  ERIC  and 
PsycLIT  (a  compact  version  of  PsycINFO) .  This  procedure  was  used 
to  determine  the  *'hit-rate”  of  each  keyword  so  that  this 
information  could  be  used  as  a  guide  in  preparing  for  the  primary 
search. 


Table  5 

Keywords  Used  in  Subsequent  Literature  Search 


Training 

Learning 

Performance 

Asymptote 

Training  and  Learning  Performance  and  Asymptote 

Skill 

Transfer 

Learning  and  Skill  Transfer 
Transfer  of  Training 
Military 

Transfer  of  Training  and  Military 
Skills 

Skills  Training  and  Military 
_  Military  and  Training _ 


Through  conversations  with  AFHRL,  it  was  suggested  that  we 
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consider  conducting  future  literature  searches  via  Crew  System 
Ergonomics  Information  Analysis  Center  (CSERIAC;  a  service 
organization  that  can  search  large  government  laboratory 
databases) .  Ultimately/  a  request  was  made  of  CSERIAC  to  conduct 
an  additional  search  of  the  OTIC  database  as  well  as  a  search  of 
the  NASA  database  using  the  new  keywords.  All  publications 
obtained  as  a  result  of  this  search  were  reviewed  and  14  articles 
were  determined  to  be  relevant.  Thus,  at  this  point  we  had 
obtained  a  total  of  19  relevant  articles  (14  new  articles 
combined  with  the  5  obtained  in  the  initial  searches) . 


Non-traditional  search  techniques.  A  final  attempt  at 
expanding  the  literature  base  was  initiated  using  non-traditional 
search  strategies  (also  known  as  "brute  force  techniques") .  In 
effect,  the  search  involved  reviewing  relevant  articles  from 
personal  sources  (e.g.,  current  journals  including  Human  Factors, 
selected  proceedings  of  the  Human  Factors  Society,  reference 
sections  of  books,  reference  sections  of  articles) .  In  addition. 
Volumes  I  and  II  of  the  Handbook  of  Perception  and  Human 
Performance  by  Boff,  Kaufman,  and  Thomas  (1986)  were  examined. 
Following  this  search,  14  prospective  articles  were  obtained  from 
local  university  libraries  and  these  articles  were  evaluated. 
After  evaluating  the  articles,  8  were  determined  to  be  relevant 
to  the  project.  Thus,  ultimately  we  were  able  to  obtain  a  total 
of  27  relevant  articles  (5  from  the  initial  on-line  search,  14 
from  the  subsequent  on-line  search,  and  8  from  personal  sources; 
see  Table  6  below  for  pertinent  characteristics  of  the  relevant 
articles) . 

As  seen  in  Table  6,  the  Information  Processing  taxonomy  and 
the  Visual  taxonomy  were  the  only  taxons  in  which  articles  were 
identified  as  being  data-rich  and  data-relevant .  It  may  be  that 
the  areas  of  Information  Processing  and  Visual  Performance  are 
research  areas  in  which  there  is  more  emphasis  on  the  interaction 
of  training  and  human  performance.  If  this  were  the  case,  then 
it  would  partly  explain  our  ability  to  obtain  more  literature  in 
these  areas.  However,  it  is  also  possible  that  more  non- 
traditional  search  techniques  might  reveal  additional  data  in  the 
Information  Processing  and  Visual  Performance  areas  as  well  as 
the  other  taxa.  For  a  complete  list  of  the  distribution  of 
relevant  articles  per  taxa,  see  Table  7. 
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Table  6 

Characteristics  of  Relevant  Articles 


TAXA 

B 

Author 
&  Date 

Independent 

Variables 

Dependent 

Measures 

INFO. 

PROC. 

Basadur 
et  al. 
(1982) 

Experiential  or 
traditional 
training  in 
creative  problem 
solving 

a)  Self-report 
attitudinal/ idea¬ 
tion 

questionnaire 

b)  Product  planning 
notes  of  subjects 
recorded  on  tape 

c)  Self-reported 
interviews  of 
subjects  observa¬ 
tions  of  job  perf. 
in  industrial 
engineering 

2 

Boreham 

(1985) 

Hypothesis  testing 
training  for 
simulated 
diagnostic  taslcs 
used  in  salt 
packaging  plant 

Number  of 
different 
categories  of 
hypotheses 
generated  during 
pretest  and 
posttest 

3 

Briggs 
et  al. 
(1965) 

a)  Interactive  vs 
isolated  trng  of ' 
team  members  in 
radar  control 

b)  Complexity  of 
radar  control  task 

c)  Length  of 
training  for  radar 
control 

Efficiency  score 
composed  of  the 
number  of 
successful 
intercepts,  time, 
and  fuel  consumed 

4 

1 

Card 
et  al . 
(1978) 

a)  Type  of  Device 
(mouse,  joystick, 
step  keys,  text 
keys) 

b)  target  size  (# 
of  letters  of 
text) 

c)  Distance  from 
starting  point  on 
all  positioning 
devices 

a)  Positioning 
speed 

b)  Homing  time 
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Table  6 

Characteristics  of  Relevant  Articles  (Continued) 


TAXA 

B 

Author 
&  Date 

Independent 

Variables 

Dependent 

Measures 

INFO. 
PROC. 
Cont . 

5 

Fisk 
et  al. 
(1988) 

a)  Memory  set  size 

b)  Possibility  of 
correct  decision 
(positive  or 
negative  trials) 

a)  Reaction  time  in 
a  dual-task 
situation 

6 

Johnson 

(1978) 

Initial  training 
strategy  for 
"checklist” 
sequential  task: 
conventional, 
reproduction,  or 
blind  strategy 

a)  Sequence  and 
setting  errors  on 
sequential  task 

b) Time  to  perform 
last  trial  of 
sequential  task 

c)  Transfer  errors 
on  actual 
sequential  task 
(e.g.,  control 
panel  equipment) 

7 

Kanfer 
et  al. 
(1989) 

a) ATC  part -task 
trng  strategy: 
declarative  or 
procedural 

b) Goal  setting 
strategy  group: 
goal  vs  no  goal 
instructions 

c) Ability  via 

ASVAB  score 

Number  of  planes 
landed  on  six 
trials  of  full  ATC 

8 

Malloy 
et  al. 
(1978) 

Cognitive  trng 
strategy  group: 
using  Raven' s 
Progressive 

Matrices  Test, 
using  brief 
exposure  to  test, 
or  no  exposure 

a)  Problem  solving 
performance  on  17 
matrix  puzzles 

b)  Piagetian 
multiplicative 
classification 
test 

9 

Masson 

(1986) 

Degree  of  exposure 
to  word  triplets 

Word 

identification 

score 
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Table  6 

Characteristics  of  Relevant  Articles  (Continued) 


TAXA 


INFO. 
PROC. 
Cont . 


10 


11 


12 


13 


14 


15 


Author 
&  Date 


Miller 

(1975) 


Morris 
et  al . 
(1985) 


Myers 
et  al . 
(1987) 


Patrick 
et  al . 
(1988) 


** Roberts 
et  al . 
(1983) 


Rouse 

(1978) 


Independent 

Variables 


Instructional 
strategy  for 
Improved  Hawk: 
traditional  vs 
supplemental 


Instructional 
training  for 
dynamic  production 
process  using 
simulator  (PLANT) 


Mapping  of  info, 
(variably  or 
consistently 
mapped)  in  trng 
for  telecomm, 
visual  search  task 


Trng  materials: 
technical  story  vs 
heuristics/  in 
fault  finding 
tasks 


Experience  with 
text  editors: 
novice  vs  expert 


a)  Problem  size  in 
fault  diagnosis 
using  graphically 
displayed  network 
system  as 
simulation 

b)  Pacing:  forced 
vs  non-forced 

c) Display: 
computer  aided  vs 
non-aided 


Dependent 

Measures 


a)  Practical  quiz 
test  score  on 
Improved  Hawk 

b)  Standardized 
score  on  Improved 
Hawk 


Production 
measured  on 
dynamic  production 
process  simulator 
(PLANT) 


a)  Reaction  time 
on  telecomm, 
visual  search  task 

b)  Accuracy  on 
telecomm,  visual 
search  task 


Accuracy  and  speed 
in  fault  diagnosis 
test  in  transfer 
scenario 


a)  Time-score  for 
each  text  editor 

b)  Individual  error 
score  for  each 
text  editor 


a)  Number  of  tests 
before  correct 
diagnosis 

b)  Percent  correct 
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Table  6 

Characteristics  of  Relevant  Articles  {Continued) 


TAXA 

« 

Author 
&  Date 

Independent 

Variables 

Dependent 

Measures 

INFO. 
PROC. 
Cont . 

16 

Scott 
et  al. 
(1982) 

Use  of  SH-3 

Aircraft  Training 
aids:  traditional 
or  supplemental 

Errors  on  cockpit 
procedures  trainer 
(CPT) 

17 

Sheppard 

(1984) 

a)  Task 

configuration; 
part  vs  whole  task 

b) FLOLS  type; 
conventional  or 
vertical  bars 
added 

c) FLOLS  size 

RMS  Glideslope 
error 

18 

Weitzman 
et  al. 
(1979) 

Type  of  Trng 
instrument:  Device 
2B24  simulator/ 
UH-IH  helicopter/ 
or  both 

Instructor  pilot 
ratings  of 
performance 

VISUAL 

1 

Coc)crell 

(1979) 

Target  quality 
during  trng; 
degraded  vs  normal 

Mean  number  of 
correct 

identifications  of 
degraded  targets 

2 

Eberts 

(1987) 

a)  Task  type: 
single  or  dual 

b)  Augmentation 
type  (point  or 
parabola  cues) 

RMS  tracking  perf. 

3 

Edward 
et  al. 
(1979) 

Pretraining: 
multimedia  vs  no 
pretraining 

Instructor-rated 
transfer  perf.  in 
the  T-37  aircraft 

4 

Lintern 
et  al. 
(1990) 

Landing 
instruction; 
supplemental 
simulator  trng  or 
traditional  trng 

Raw  tracking  error 
on  landing  task 
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Table  6 

Characteristics  of  Relevant  Articles  (Continued) 


TAXA 

o 

Author 
&  Date 

Independent 

Variables 

Dependent 

Measures 

VISUAL 
Cont . 

5 

* Rabbit 
et  al. 
(1979) 

Length  of  practice 
on  visual  search 
of  embedded 
letters 

Reaction  time  for 
visual  search  of 
target  letters 

6 

Schneider 
et  al. 
(1984) 

a)  Exp.  1:  Type  of 
mapping:  variably 
vs  consistently 
during  trng  for 
category  search 
task 

b) Exp.  2:  Number 
of  concurrent 
tasks:  Single  or 
dual 

a)  Exp.  1:  Visual 
search  reaction 
time  improvement 

b) Exp.  2:  Percent 
improvement  of 
word  detection 

7 

Simon 
et  al. 
(1981) 

Simulator 
configurations 
(variations  of : 
control  order, 
display  lag, 
tracking  mode, 
prediction  time, 
control  gain,  and 
number  of  trials) 

Tracking  error  as 
measured  on  the 
Visual  Technology 
Research  Simulator 

8 

Wightman 
et  al. 
(1987) 

a)  Trng  task:  whole 
vs  segmented  task 

b)  Carrier  landing 
lag:  normal  lag  vs 
progressive  lag 

RMS  carrier 
landing  as 
measured  on  the 
Visual  Technology 
Research  Simulator 

9 

Wrisberg 
et  al. 
(1983) 

Trng  condition: 
open  skill 
(requiring  varying 
reactions)  vs 
closed  skill 
(requiring  stable 
reactions) 

Absolute  error  of 
arm  movement 
during  transfer 

mJTF^TnTes^^arTteJ^a  stiown^^iowT^S^fulle^^ntain  o^^^roup  data 
•  Study  contains  only  individual  data 

**  study  contains  botfs  individual  and  group  data 
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Table  7 

Distribution  of  Relevant  Articles  Obtained  Per  Taxa 


DISTRIBUTION  OF  RELEVANT  ARTICLES 
OBTAINED  PER  TAXA 

Information  Processing 

18 

Numerical 

0 

Visual 

9 

Fine  Motor  (Discrete) 

0 

Fine  Motor  (Continuous) 

0 

Gross  Motor  (Heavy) 

0 

Gross  Motor  (Light) 

0 

Communication  (Read/Write) 

0 

Communication  (Oral) 

0 

TOTAL 

27 

A  textual  and  graphical  summary  of  each  of  the  27  relevant 
articles  can  be  found  in  Appendix  A.  A  list  of  citations 
including  only  these  27  relevant  articles  can  be  found  in 
Appendix  B.  In  effect,  Appendix  B  is  intended  as  a  bibliography 
of  articles  contained  in  Appendix  A.  For  an  annotated  listing  of 
articles  obtained  and  reviewed  during  all  the  searches,  see 
Appendix  C.  Appendix  C,  thus,  is  intended  as  a  bibliography  of 
all  publications  that  were  accessed  through  the  traditional 
keyword  as  well  as  the  non-traditional  searches. 

Over  the  course  of  searching  the  literature  for  publications 
specifically  examining  the  quantitative  relationships  between 
training  and  soldier  performance,  it  became  apparent  that  the 
traditional  keyword  search  technique  was  ineffective.  The 
keyword  approach  allows  one  to  easily  obtain  traditional 
literature  pertaining  to  learning  as  well  as  traditional 
literature  pertaining  to  human  performance  and  training. 

However,  it  appears  that  this  approach  does  not  effectively  lend 
itself  to  directly  accessing  literature  common  to  all  of  these 
areas.  This  is  evidenced  in  the  fact  that  more  publications  were 
obtained  via  non-traditional  search  strategies  than  in  the 
initial  keyword  search. 
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r«atlbillty  of  Dovoloping  a  Stand  Alona  Human 
Raliabillty  Growth  Rodal 


The  overall  feasibility  of  developing  a  stand  alone  HRGM 
depends  upon  satisfying  the  following  issues: 

1.  Do  sufficient  quantities  of  suitable  data  exist? 

2.  Does  an  adequate  theoretical  basis  exist  for  developing 
such  a  model,  data  adequacy  aside? 

3.  Is  a  stand  alone  quantitative  model  feasible,  or  should 
the  resultant  algorithms  be  incorporated  within  the 
structure  of  a  human  performance  simulation  model? 

This  chapter  addresses  each  issue  in  turn.  The  first  section  of 
this  chapter  addresses  issue  #1.  The  criteria  for  the  adequacy 
of  actuarial  data  presented  at  the  conclusion  of  Chapter  2  are 
applied  to  this  question.  The  middle  section  of  this  chapter 
addresses  issue  #2.  The  body  of  knowledge  regarding  human 
learning  and  practice  and  the  learning  curve  construct  is 
revisited  to  address  the  nature  of  a  HRGM.  The  final  section  of 
this  chapter  addresses  issue  #3.  Given  the  resolution  of  the 
first  two  issues,  recommendations  are  addressed  for  the 
development  of  a  stand  alone  quantitative  model  versus 
incorporation  within  ongoing  efforts  to  develop  human  performance 
simulation  models.  An  Illustration  of  a  'successful'  algorithm 
IS  also  presented  in  order  to  describe  the  ideal  situation. 

Do  Sufficient  Quantities  of  Suitable  Data  Exist? 

The  issue  which  predominates  throughout  this  project  is  the 
availability  of  adequate  data.  Absent  the  ability  to  collect  new 
data  depicting  the  relationship  of  training  variables  to  human 
performance,  we  must  depend  upon  the  existence  of  acceptable 
actuarial  data  in  the  reported  behavioral  and  social  science 
literature  to  provide  these  data.  The  criteria  presented  in 
Table  3  describe  the  basis  for  determining  the  acceptability  of 
the  available  data.'  Table  8  summarizes  our  findings  with  respect 
to  our  analysis  of  the  data  base  assembled  through  the  literature 
reviews  conducted  during  this  project  (as  described  in  the 
previous  chapter) .  The  paragraphs  that  iollow  discuss  each 
criterion  in  turn. 
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Table  8 

Summary  of  Findings  Regarding  the  Adequacy  of  Actuarial  Data  for 
the  Development  of  a  Stand  Alone  HRGM 


Criteria  for  Kvaluating  the 
Adequacy  of  Actuarial  Data 

Suamary  of  Findings  with 
raspaet  to  Analysis  of  Data 
Contained  in  Sxtant  Literature 

1.  Existence  of  actuarial  data 

•  27  research  articles 

identified  which  contain 
data  on  variables  of 
interest 

a.  Measurement  of  Predictor 

&  Criterion 

•  27  research  articles 

containing  clearly 
identified  criterion  & 
predictor  relationships 

b.  Preservation  of 

individual  differences  or 
of  meaningful  groups  of 
subjects 

•  26  research  articles 
presenting  group  data' 

•  1  research  articles 
presenting  individual 
data 

c.  Comparability  of 

independent  &  dependent 
variables  within  a  taxa 

•  Wide  variation  in  types 
of  both  independent 
measures  &  dependent 
measures  within  available 
taxa 

•  Pooling  of  data  within 
taxa  not  feasible 

d.  Same  (or  transformable  to 

same)  independent  & 
dependent  measures  across 
studies 

•  Indspondont  moasuros : 

0%  comparable 

100%  not  comparable 

•  Dapsndont  aaasuras : 

26%  comparable 

74%  not  comparable 

Wote:  One  article  contained  both  individual  and  group  data. 
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Criteria  for  IvaXuatlng  tho 
Adoquaoy  of  Actuarial  Data 

Suaaary  of  Flndlnga  with 
raspoot  to  Analysis  of  Data 
Contained  in  Extant  Litaratura 

e.  Sufficient  sample  size 

• 

Sample  size  breakdown: 

M  <  10:  26%  of  research 
articles 

H  >  10  and  <  50:  48%  of 

research  articles 

N  >  50:  26%  of  research 
articles 

• 

Small  sample  sizes 
prevalent  which 
undermines  stability  of 
the  learning/performance 
relationships  contained 
in  the  literature 

• 

Insufficient  stability  in 
learning  curves  for  I 

projected  population  1 

(Army  soldiers) 

2.  Relevance  to  military  tasks 

• 

15%  relevant  to  Army 
tasks 

• 

26%  relevant  to  Military 
tasks  (other  than  Army) 

• 

59%  not  relevant  to 
military  tasks 

3.  Relevance  to  military 
training 

• 

15%  relevant  to  Army 
training 

• 

26%  relevant  to  Military 
training 

# 

59%  not  relevant  to 
military  training 

4.  Comprehensiveness  of  Taxa 
Coverage 

Within :  Indeterminate , 
but  believed  inadequate 
for  the  2  taxa  covered 

• 

Aoroas:  Coverage  of  2  of 

9  broad  taxa  contained  in 
the  PER-SEVAL  model 
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Existence  of  Actuarial  Data 


A  total  of  27  research  articles  was  selected  out  of  a  list 
of  approximately  3,000  articles  identified.  These  27  present 
learning  data  of  the  type  required  for  this  project.  Eighteen  of 
the  research  articles  were  identified  as  belonging  to  the 
information  processing/problem  solving  taxon.  Nine  of  the 
research  articles  were  identified  as  belonging  to  the  visual 
recognition/discrimination  taxon.  No  research  articles  whatever 
were  identified  for  the  remaining  seven  taxa.  This  compilation 
of  literature  represents  a  broad  range  of  independent  and 
dependent  variables.  As  a  consequence,  no  obvious  classification 
structure  was  evident  as  a  basis  for  pooling  studies  within  a 
taxa.  In  sum,  while  data  exist,  paucity  and  diversity  in  the 
range  of  variables  encompassed  overwhelm  potential  usefulness  for 
the  development  of  a  stand  alone,  quantitative  model.  The 
following  summari2es  the  results  obtained  in  terms  of  five 
subcriteria. 

(1)  Measurement  of  Predictor  and  Criterion.  In  general, 
the  research  contained  in  Appendix  A  should  be 
classified  as  'near  misses',  that  is,  data  are  reported 
within  each  article  which  may  be  used  to  approximate 
both  criterion  and  predictor  data.  Predictor  data 
(i.e.,  training  data)  uniformly  exist  for  the 
literature  reported.  However,  criterion  data  (i.e., 
job  performance  data)  must  be  approximated,  because 
they  are  not  generally  presented  as  such.  Proficiency 
(the  ability  to  perform  a  task  that  has  been  trained 
through  a  'job  sample'  or  simulated  performance)  is 
provided  as  the  criterion.  Job  performance  data 
require  the  availability  of  'external'  measures  of 
criterion  performance  (meaning  external  to  the  training 
environment) .  As  a  consequence,  the  utility  of  the 
data  must  be  weighed  against  the  modeler's  acceptance 
of  'proficiency'  data  as  'job  performance'  criterion 
measures . 

(2)  Preservation  of  individual  difference  or  of  meaningful 
groups  of  subjects.  As  Lane  (1986b)  maintains,  there 
are  differences  between  the  use  and  interpretation  of 
learning  curves  based  on  individual  versus  group  data. 
Only  two  of  the  research  articles  contained  in  Appendix 
B  contain  individual  learning  data.  Twenty-six  of  the 
articles  present  group  data.*  One  article  reports 
only  group  data.  Individual  data  are  preferred  since 
these  provide  the  best  basis  for  determining  the 
variability  attendant  to  task  learning  within  a  group. 


’  NOTE:  Ore  article  reports  both  individual  and  group  data. 
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Given  that  few  such  data  were  obtained  in  the 
literature  search,  we  are  dependent  upon  the  use  of 
group  data.  We  have  left  the  question  of 
'meaningfulness'  to  tha  user  of  the  data,  since  this  is 
a  subjective  metric  dependent  upon  the  context  of  use 
and  the  interpretations  of  'meaningfulness'  by  the 
analyst.  The  only  criterion  used  in  selecting  the 
research  articles  to  include  was  that  the  subject 
population  represent  a  'normal  adult  population'  of 
subjects.  Clearly,  if  'meaningfulness'  involves 
separate  treatment  groups  distinguished  by  aptitude 
then  the  data  reported  here  do  not  apply.  However,  if 
'normal  adult  population'  is  considered  representative 
of  the  'military  population'  then  dependent  upon  the 
situation,  the  data  may  apply.  This  question  is  for 
the  analyst  using  the  data  to  decide. 

(3)  Comparability  of  independent  and  dependent  variables 
within  a  taxa.  In  order  to  pool  data^  of  a  single 
taxa  reported  in  this  literature,  the  data  must  have 
comparable  independent  and  dependent  variables.  That 
is,  they  must  contain  and  vary  the  same  training  and 
job  performance  variables.  As  shown  in  Table  6,  the 
literature  contains  a  broad  range  of  independent 
variables,  including: 

•  type  of  training, 

•  complexity  of  training, 

•  training  materials, 

•  learning  strategy,  and 

•  extent  or  amount  of  training/practice. 

These  disparate  training  variables  lack  a  common  basis 
for  classification  in  order  to  permit  ready  pooling  of 
data.  Likewise,  Table  6  contains  a  broad  range  of 
dependent  variables,  including: 

•  Problem  solving, 

•  Transfer  of  skill, 

•  Spatial  rule  learning, 

•  Problem  solving  quiz, 

•  Learning  score, 

•  Simulator  performance,  and 

•  Tracking  and  visual  discrimination 

performance . 

The  dependent  variables  also  lack  a  common  basis  for 
pooling.  No  two  studies  have  both  comparable 


*  Pooling  of  data  appears  necessary  In  order  to  obtain  the  most  robust  and 
comprehensive  basis  for  developing  and  applying  learning  curves  for  use  in  a  model. 
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independent  and  dependent  variables  on  which  to  pool 
data. 

(4)  Same  (or  transformable  to  same)  independent  and 

dependent  measures  across  studies .  Even  in  cases  where 
the  variables  are  the  same  (or  similar)  the  measures 
used  to  operationalize  the  variables  must  also  be  the 
same  (or  similar)  in  order  to  pool  the  data  across 
studies  within  a  taxa.  Similar  training  or  performance 
variables  identified  in  the  literature/  lac)c  similar 
operational  measures.  For  example/  in  the  case  of 
independent  variables,  the  number  of  training  trials  is 
not  readily  transformable  to  training  time.  (Such  a 
conversion  would  require  Icnowledge  of  time  elapsed  per 
trial.)  In  the  case  of  dependent  measures, 
operationalized  measures  of  time  or  accuracy  are 
dissimilar  from  one  another.  Laclcing  a  common 
yardstic)c  of  comparison  results  in  an  inability  to  pool 
data  within  a  taxa. 


(5)  Sufficient  sample  size.  Two  training  variables  (known 
as  Performance  Shaping  Functions  or  PSFs)  were  included 
in  the  PER-SEVAL  report  (O'Brien  et  al.,  1989).  These 
variables  are  frequency  and  recency  of  training.  The 
sample  size  for  each  of  these  variables  varied  from 
Taxa  to  Taxa  and  tasks  within  taxa,  but  the  range  was 
from  346  to  a  high  of  694  soldiers  per  task.  Most  were 
above  600  soldiers  as  the  sample  size.  Only  two  of  the 
studies  described  in  Appendix  A  reported  samples 
anywhere  near  this  magnitude.  One  study  reported  a 
sample  size  of  552  subjects  and  a  second  reported  288 
subjects.  Studies  with  these  sample  sizes  are  in  the 
clear  minority.  The  range  in  sample  size  is  shown  in 
Table  9.  As  can  be  observed  from  the  table,  74%  of  the 
articles  reported  subjects  with  no  better  than  moderate 
sample  sizes.  This  would  tend  to  undermine  the 
generalizability  of  the  data  to  other  subject 
populations.  In  sum,  the  reported  literature  does  not 
generally  approach  the  sample  sizes  used  in  similar 
research  efforts. 
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Table  9 

Range  of  Sample  Sizes  Across  Selected  Research  Articles 


RANGE  in  #  of  Subject 

Nuaiber  (and  Parcantaga) 
in  Range 

Low  (<  10  subjects) 

7  articles  (26%) 

Medium  (26  -  53 
subjects) 

13  articles  (48%) 

High  (54  -  552) 

7  articles  (26%) 

Relevance  to  Military  Tasks 

Relevance  to  military  tasks  is  an  important  issue  (as  noted 
by  Lane,  1986b) .  Research  on  tasks  which  are  not  directly 
relevant  to  military  tasks  and  environments  may  not  be 
generalizable  to  military  situations.  Some  tasks  performed  in  a 
non-military  setting  may  have  no  military  counterpart.  Of  the  27 
research  articles  contained  in  Appendix  A,  less  than  50% 
represent  military-based  research.  A  simple  majority  of  the 
research  comes  from  other  than  military  environments.  While  this 
does  not  immediately  result  in  an  inability  to  generalize  these 
data,  it  does  raise  the  concern  of  generalizability .  Relevance 
is  determined  by  subject  matter  expert  judgement  and  is  not 
easily  assessed  in  any  other  manner.  Even  if  given  researchers 
are  willing  to  accept  research  findings  obtained  out  of  the 
military  context,  other  users  of  a  HRGM  may  question  the  model's 
validity  on  this  basis. 

Relevance  to  Military  Training 

In  like  manner  to  the  above  criterion,  data  on  specific 
training  variables  not  obtained  in  a  military  training 
environment  may  not  adequately  generalize  to  the  military 
situation  for  two  reasons; 

•  the  'treatments'  given  to  training  may  not  be  feasible 
in  the  military  training  context  (e.g.,  training  time 
in  the  military  is  more  a  question  of  budget  and 
tradition  -  that  is,  how  much  training  was  provided  in 
the  past  for  this  or  a  similar  task;  in  other  non¬ 
military  training  situations  these  constraints  may  not 
exist) ,  and 

•  the  training  variable  may  have  no  practical  alternative 
in  the  military  context. 
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The  percentages  of  articles  relevant  to  military  tasks  apply 
equally  well  to  military  training  relevance.  In  sum,  relevance 
to  military  tasks  and  training  must  both  be  judged  and  the 
effects  of  acceptance  by  model  users  must  be  considered  in  these 
judgements . 

Comprehensiveness  of  Taxa  Coverage 

The  degree  to  which  a  given  set  of  data  or  learning  curve 
relationships  depict  the  'true'  relationship  of  training 
variables  to  performance  within  a  given  taxa,  is  in  part  a 
question  of  the  degree  to  which  the  data  are  representative  and 
comprehensive  of  the  relationships  theoretically  possible  in  each 
taxa.  For  this  project,  this  is  indeterminate,  because  we  simply 
don't  know  what  we  don't  know.  This  indeterminacy  creates 
problems  particularly  for  sets  of  learning  curve  relationships 
that  are  based  on  small  or  only  moderate  sample  sizes.  The 
concern  with  the  underlying  generalizability  of  the  data  to  a  new 
context  is  proportionally  increased.  We  not  only  do  not  know  how 
generalizable  the  data  are,  we  also  don't  know  how  much  error  we 
may  introduce  as  a  consequence  of  using  the  data. 

Across  the  nine  taxa  in  which  sources  of  data  were  sought, 
only  two  of  the  nine  taxa  yielded  any  reported  findings. 

Clearly,  the  comprehensiveness  of  coverage  of  the  range  of  taxa 
found  in  the  military  settings  is  partial  at  best.  Any  stand 
alone  model  which  could  be  developed  should  contain  at  least  some 
data  for  each  of  the  relevant  taxa.  This  clearly  was  not  the 
case  here. 


Probable  Reasons  for  a  Lack  of  Suitable  Data  Reported  in  the 
Behavioral  and  Social  Science  Literature 

The  lack  of  suitable  data  which  would  permit  the  development 
of  a  quantitative  stand  alone  HRGM  was  disappointing.  During  the 
course  of  the  literature  reviews,  reports  containing  similar 
objectives  to  our  own  were  reviewed.  Their  findings  were  much 
the  same  as  our  ov/n.  That  is,  the  theoretical  basis  for 
developing  and  employing  learning  curve  fitting  techniques  for 
evaluating  the  relationship  of  training  variables  to  human 
performance  is  adequate.  The  data  base  supporting  such  an  effort 
does  not  exist  at  this  time  (see  Lane,  1986b  and  Kennedy  et  al., 
1988a;  1988b  for  a  detailed  report  of  these  findings) .  Table  10 
presents  a  summary  of  the  probable  explanations  for  the  lack  of 
suitable  data  reported  in  the  behavioral  and  social  science 
literature.  We  can  offer  no  speculation  as  to  the  relative 
frequency  of  occurrence  of  any  of  these  probable  reasons  for  the 
failure  of  the  literature  to  support  our  data  requirements.  This 
failure  may  simply  reflect  the  difficulties  and  expense  involved 
in  collecting  data  of  the  type  required  by  this  project. 
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Table  10 

Probable  Reasons  for  Lack  of  Suitable  Data 


• 

Empirical  studies  of  human  learning  do  not  typically 
involve  predictions  to  an  external  criterion  (i.e.,  job 
performance) 

• 

Initial  learning  or  training  apparently  has  not  had  wide 
application  as  a  predictor  in  industrial/organizational 
psychology 

• 

Development  of  good  criterion  measures  of  job 
performance  are  difficult  to  secure  for  a  variety  of 
practical  or  theoretical  reasons  and  intermediate 
measures  of  proficiency  (results  of  post-training  job 
knowledge  tests)  are  more  convenient  to  acquire  as  a 
substitute 

• 

Research  that  produces  no  significant  findings  tends  not 
to  be  reported  in  the  literature 

• 

Lack  of  opportunity  or  support  resources  to  collect 
training  and  more  particularly  performance  data  (e.g., 
no  opportunity  to  manipulate  training  variables  such  as 
an  invariant  training  schedule/  lack  of  control  over 
trials  and  practice  periods) 

• 

Lack  of  concern  among  researchers 

Does 

The  'ubiquitous'  learning  curve  appears  to  provide  a  solid 
technical  basis  for  the  development  of  a  HRGM.  The  summary  of 
documented  research  on  the  learning  curve  provided  in  Chapter  2 
demonstrates  the  veracity  of  the  learning  curve  as  a  driving 
construct  in  the  development  of  a  HRGM.  What  remains  is  to 
describe  the  manner  in  which  the  learning  curve  might  best  be 
employed  within  a  stand  alone  model.  It  is  not  appropriate  to 
state  the  exact  fcrm  that  such  a  model  should  take  with  any 
finality.  This  statement  is  based  on  the  following  observations 
from  our  review  of  the  literature: 

1.  First  and  foremost,  learning  curves  are  data  driven. 

That  is,  the  exact  form  of  the  learning  curve  function 
is  derived  directly  from  the  data  themselves.  Until 
such  time  that  sufficiently  robust  data  are  produced, 
the  search  for  the  truly  generalizable  learning  curve 
applicable  to  specific  behavioral  taxonomic  elements 
appears  to  be  a  search  for  the  'holy  grail' .  A  broadly 
generalizable  learning  curve  function  appears  to  be  the 
central  requirement  of  such  a  model  of  human 
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reliability  growth. 


2 .  There  is  no  consensus  on  the  'optimum'  form  of  the 
learning  curve  function  (i.e./  power,  hyperbolic, 
exponential,  logistic) .  Depending  on  the  situation, 
each  function  has  been  demonstrated  as  'the  best  fit' 
equation . * 

3.  The  four  specific  parameters  of  the  learning  curve 
identified  by  Spears  (1983) ;  beginning  level  learning, 
asymptote,  rate  of  learning,  and  inflection  point 
(discussed  in  Chapter  2)  are  of  value  to  the  prediction 
of  differing  training  variables  -  depending  upon  the 
circumstances.  For  example,  asymptote  is  most  useful 
in  determining  the  optimum  training  time  or  practice 
trials  for  the  learning  of  tasks.  Rate  of  learning  is 
also  useful  in  this  determination,  but  its  focus  is 
towards  the  determination  of  'what  happens  to  task 
proficiency  if  I  stop  trials  here'  rather  than  the 
determination  of  optimum  performance  levels  based  on 
the  achievement  of  asymptote  levels  of  proficiency. 

The  other  parameters  have  value  in  model  applications 
as  identified  in  Chapter  2. 

4.  The  nature  of  task  performance  in  the  complex  settings 
of  military  systems  may  overly  complicate  our  ability 
to  predict  the  relationship  of  training  variables  on 
job  performance.  Tasks  which  contain  multi-component 
operations  may  'be  more  than  the  sum  of  their  parts' 
when  applying  current  learning  curve  technology.  This 
research  is  still  largely  reductionistic  and  we  lack 
the  capability  to  integrate  the  effects  of  numerous 
training  variables  (e.g.,  time,  media)  on  the 
performance  of  tasks  containing  multi -component 
operations . 

Recognizing  these  limitations,  we  can  nevertheless 
illustrate  the  desired  characteristics  and  operation  of  an 
'idealized'  algorithm  to  which  the  HRGM  should  aspire.  What 
follows  is  a  description  of  an  'idealized'  example,  that  is,  the 
'heat  effects'  algorithms  used  by  Siegel,  Wolf,  and  Bartter 
(1982)  . 

The  'heat  effects'  algorithms  developed  by  Siegel  et  al. 
(1982)  were  used  in  adjusting  the  impact  of  heat  on  human 
abilities  within  the  Maintenance  Personnel  Performance  Simulation 
(MAPPS)  developed  by  the  same  authors.  In  nuclear  power  plant 


‘  This  may  in  fact  constitute  an  academic  discussion,  since  there  is  no  inherent 
reason  that  the  HRGM  must  be  based  on  a  'single'  best  fit  equation.  It  simply  means  that 
we  cannot  lay  out  the  'idealized'  HRGM  with  any  degree  of  finality  at  this  time. 
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operations  (as  In  other  environments)/  heat  has  significant 
impacts  upon  the  performance  (both  physical  and  cognitive)  of 
humans.  High  heat  conditions  occur  in  situations  when  the 
external  temperature  increases  beyond  the  comfort  and/or 
tolerance  ranges  for  humans  performing  'perishable'®  tasks. 
Further,  it  is  known  that  heat  stress  causes  incrementally 
greater  decrements  in  task  performance  requiring  mental  or 
cognitive  operations  than  for  physical  operations  (Siegel  et  al., 
1982) .  We  should  anticipate,  then,  that  cognitive  task 
performance  of  humans  will  experience  high  levels  of  stress, 
reduced  capability  to  perform  under  conditions  of  high  workload, 
greater  response  time,  and  reduced  overall  accuracy  in  task 
performance  when  heat  stress  is  a  factor.  Any  model  which 
purports  to  be  inclusive  of  the  major  variables  that  affect  human 
performance  in  which  heat  is  expected  to  play  a  significant  role 
should  be  expected,  therefore,  to  be  sensitive  to  differences  in 
task  performance  that  occur  as  a  function  of  heat  stress. 

Siegel  et  al.  (1982)  derived  mathematical  relationships  from 
the  isodecrement  functions  presented  in  Figure  4  (A  &  B)  in  order 
to  model  the  effects  of  heat  stress  on  maintenance  task 
performance  in  nuclear  power  plants.  They  applied  the  results  of 
two  literature  reviews  (Grether,  1973;  Ramsey  &  Morrissey,  1978) 
that  consolidated  the  research  findings  related  to  the  effects  of 
heat  on  human  performance.  These  studies  summarized  the  findings 
of  16  cognitive  studies,  53  reaction  time  and  cognitive  studies, 

7  vigilance  studies,  14  tracking  studies,  and  48  perceptual-motor 
studies  in  which  heat  was  an  independent  variable.  The 
integration  of  the  results  of  these  studies  resulted  in  the 
isodecrement  curves  contained  in  the  figures.  Figure  4  (A)  plots 
the  effect  of  temperature  over  time  on  task  performance  for 
mental  reaction  time.  Figure  4  (B)  plots  the  effect  of 
temperature  over  time  on  task  performance  combining  tracking, 
vigilance,  and  complex  tasks.  From  these  plots,  two  regression 
equations  were  derived,  one  for  intellective  tasks  and  another 
for  perceptual-motor  tasks.  These  equations  are  shown  in  Table 
11.  These  equations  were  then  used  to  develop  ability  decrement 
functions  for  maintainer  performance  as  a  function  of  temperature 
over  time,  as  .shown  in  Table  12.  The  functions  were  then 
incorporated  within  the  ability  decrement  subroutines  programmed 
for  the  MAPPS  model  to  calculate  ability  decrements  whenever  heat 
is  included  as  an  input  variable  in  the  simulation. 


’  Perishable  in  the  sense  that  task  performance  degrades  as  a  direct  consequence  of 
heat  stress  effects  on  humans. 
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Figure  5  illustrates  the  logic  flow  that  was  used  to  calculate 
ability  decrements  as  a  function  of  heat  stress  in  maintenance. 

During  a  MAPPS  model  application,  the  user  designates  the 
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variables  which  would  impact  performance  by  selecting  'HEAT 
STRESS'  as  a  variable  to  incorporate  into  the  model  runs.  Over 
time,  heat  stress  would  have  an  accumulative  effect  on  the 
ability  level  (and  other  workload  indications)  of  maintenance 
performance.  The  MAPPS  model  calculates  the  ability  decrement  of 
individuals  using  the  heat  decrement  subroutine  (depending  upon 
the  kind  of  task  being  simulated  at  that  point  in  the  model 
(i.e.,  intellective  or  perceptual -motor  tasks  in  the  current 
example) . 


TIME  ON  TASK 

TEMPERATURE 


Figure  5.  Ability  decrement  for  heat  stress  logic  flow. 


In  the  current  context,  it  is  clear  that  successful 
prediction  of  the  effects  of  heat  stress  on  performance  (defined 
in  this  instance  in  terms  of  intellective  and  perceptual-motor 
ability  decrements) ,  depends  upon  the  very  same  criteria  of  data 
adequacy  that  were  used  in  evaluating  the  learning  data  gleaned 
from  the  research  literature  for  this  study.  That  is: 

•  The  data  required  to  support  the  development  of  the 
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Isodecrement  curves  for  the  effects  of  heat  stress  on 
performance  obviously  existed  in  sufficient  quantities 
(16  cognitive  studies,  53  reaction  time  and  cognitive 
studies,  7  vigilance  studies,  14  tracking  studies,  and 
48  perceptual-motor  studies) . 

•  In  addition,  the  same  (or  similar)  independent  and 
dependent  variables  and  measures  were  applied,  as  can 
be  observed  from  scanning  the  scales  around  which  the 
isodecrement  functions  contained  in  Figure  4  are  based. 

•  While  the  sample  size  of  each  individual  study  are  not 
reported,  the  pooling  of  data  across  like  studies  has 
the  effect  of  increasing  the  sample  size  and  provides 
added  assurance  of  stability  in  the  reported 
isodecrement  relationships. 

•  Relevance  to  the  target  population  is  an  open  question, 
since  the  original  subject  samples  are  not  described. 
However,  performance  of  cognitive  and  perceptual-motor 
tasks  under  conditions  of  heat  stress  are  clearly 
context-relevant  taxa  (i.e.,  tasks)  to  either  the 
nuclear  power  plant  maintenance  or  military 
environments . 

•  Finally,  the  accumulation  of  the  findings  on  such  a 
discrete  topic  (heat  stress  effects  on  performance) 
from  over  100  separate  research  articles  certainly 
underscores  a  position  that  the  reported  literature  is 
comprehensive  with  respect  to  coverage  of  human 
behavior  attendant  to  that  taxa. 

In  sum,  it  can  be  categorically  stated  that  the  data  used  by 
Siegel  et  al.  (1982)  to  construct  their  heat  stress/ability 
decrements  algorithms  were  perfectly  suited  to  their  use  in  the 
algorithms . 

Data  of  a  similar  caliber  were  not  found,  as  discussed  in 
the  opening  section  of  this  chapter.  Were  this  situation  to 
change  (such  as  in  the  situation  that  new  data  relating  the 
effects  of  training  variables  to  performance  were  collected) , 
then  learning  curve  fitting  techniques  or  the  isoperformance 
methodology  developed  by  Kennedy  et  al.  (1988a,  1988b)  (described 
in  Chapter  2)  might  well  be  successfully  applied  to  these  kinds 
of  analyses. 


Is  a  Stand  Alone  Quantitative  Model  Feasible? 

We  can  conclude  based  on  the  prior  discussion  that  even 
though  an  adequate  theoretical  basis  for  the  development  of  a 
HRGM  probably  exists,  the  extant  data  reported  in  the  behavioral 
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and  social  science  literature  do  not  exist  in  adequate  quantities 
to  support  the  development  of  a  completely  stand  alone, 
quantitative  model  as  set  out  in  this  project's  initial  goals. 
This  then  begs  the  question  as  to  what  might  be  done  with  the 
data  collected  (and  reported  as  Appendix  A) .  It  is  recommended 
that  the  data  shown  in  Appendix  A  be  considered  for  use  in  the 
development  of  algorithms  to  be  incorporated  into  an  appropriate 
human  performance  simulation  model.  The  extent  to  which  this  is 
feasible,  depends  upon;  (a)  model  requirements,  and  (b)  the 
degree  to  which  model  users  are  concerned  with  error  of 
prediction  and/or  have  the  means  to  build  statistical  control 
over  such  error  into  a  simulation  model.  While  the  same 
limitations  regarding  the  utility  of  the  data  apply  to  a  human 
performance  simulation  model  as  they  do  to  a  stand  alone 
quantitative  model,  the  simulation  model  may  be  less  affected  by 
these  sources  of  error  than  a  stand  alone  model.  As  a 
consequence,  the  criteria  and  the  results  of  our  evaluation  of 
these  criteria  apply  to  the  potential  application  of  the  data 
contained  in  Appendix  A  as  well.  The  following  list  summarizes 
the  most  serious  concerns  which  potentially  affect  the  utility  of 
the  reported  data  within  a  computer  simulation  model: 

•  generalizability  of  the  data  and  learning  curve 
relationships  to  other  contexts, 

•  stability  of  the  relationships  given  the  available 
sample  size;  representativeness  of  the  relationships 
contained  in  the  data  to  that  in  each  taxa  at  large; 
and  relevance  to  the  military  (tasks  and  training) ,  and 

•  usability  of  the  data  given  the  amount  of  data  reported 
in  the  literature. 


Conclusions  and  Recommendations 

This  study,  like  previous  studies  (e.g..  Lane,  1986b; 

Kennedy  et  al.,  1988a;  1988b;  Spears,  1983),  has  come  to  similar 
conclusions  regarding  the  theory  and  data  available  to  develop 
models  of  human  learning  and  practice.  That  is,  the  theoretical 
basis  for  the  development  of  a  model  (or  models)  using 
generalizable  learning  curves  appears  acceptable.  What  is 
missing  is  a  suitable  literature  and  data  base  on  which  to 
establish  such  a  model.  Therefore,  a  concerted  data  collection 
effort  is  required  in  order  to  establish;  (a)  the  training  and 
performance  variables  of  interest,  (b)  the  performance  measures 
and  treatment  groups  required  to  collect  appropriate  data  on 
these  variables,  and  (c)  the  collection,  reduction,  and  storage 
of  the  data  identified  with  each  of  these  steps.  Once  collected, 
the  development  of  a  HRGM  can  become  a  relatively  rapid  exercise. 

If  said  data  were  to  be  collected  as  recommended,  additional 
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research  would  be  required  to  address  known  issues  attendant  to 
the  prediction  of  the  effects  of  human  learning  and  practice. 
These  Include 

•  measurement  and  prediction  of  military  tasks  containing 
multi-component  operations, 

•  methodology  for  increasing  the  generalizability  of 
learning  curve  relationships  (e.g,,  use  of  individual 
versus  group  data;  methodology  for  increasing  the 
comparability  of  independent  and  dependent  variables 
and  measures),  and 

•  methodology  for  quantifying  the  effects  of  moderator 
variables  (e.g.,  aptitude,  job  aids,  equipment  design 
features)  that  mitigate  the  effects  of  human  learning 
and  practice  on  job  performance. 

Finally,  it  is  recommended  that  the  data  contained  in  Appendix  A 
be  examined  for  potential  utility  in  developing  algorithms  for 
use  in  a  human  performance  simulation  model. 
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APPENDIX  A 


DESCRIPTION  OF  RELEVANT  ARTICLES 


The  purpose  of  Appendix  A  is  to  present  general  descriptions 
of  the  relevant  publications  obtained  in  the  various  literature 
searches.  For  each  publication,  a  textual  description  is 
provided  that  includes  the  following  information:  bibliography, 
type  of  task  examined  in  the  study,  taxonomy  of  task,  type  of 
data  obtained  in  the  study,  subjects  who  participated,  design  of 
the  study,  type  of  training  implemented,  predictor  variable (s), 
criterion  variable (s),  and  abstract.  Following  the  textual 
descriptions  of  each  study  are  displays  of  the  relevant  data  in 
graph  and/or  table  format.  Taken  together,  the  descriptions  in 
Appendix  A  should  provide  the  reader  with  a  means  of  rapidly 
obtaining  pertinent  general  and  specific  information  contained  in 
the  studies . 
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DESCRIPTIONS  OF  RELEVANT  PUBLICATIONS 


Basadur,  M.,  Graen,  G.B./  &  Green,  S.G.  (1982).  Training  in 

creative  problem  solving:  Effects  on  ideation  and  problem 
finding  and  solving  in  an  industrial  research  organization. 
Organizational  Behavior  and  Human  Performance.  30 ,  41-70. 


Task:  Creative  problem  solving 
Taxa:  Problem  solving 
Type  of  Data:  Group 
Subjects:  45  engineers 

Design:  Simple  measurement,  no  repeated  measures 
Training : 

1.  Subjects  received  either  a  treatment  consisting  of 
training  in  a  two  day  intensive  creative  problem 
solving  condition,  they  received  a  placebo  treatment, 
or  they  received  no  treatment  at  all. 

Independent  Variable  (s) : 

1.  Experiential  training  in  creative  problem  solving 
Dependent  Measure (s): 

1.  Self-report  attitudinal /ideation  questionnaire 

2 .  Product  planning  notes  made  by  subject  verbally 
via  tape  recorder 

3.  Recorded  self-reported  interviews  detailing 
subject's  observations  of  on  the  job  creative  problem 
solving  performance  in  industrial  engineering 
organization 

Abstract:  This  study  examined  the  effects  of  multistage 
creative  problem  solving  on  attitude  and  behaviors  of 
subjects  assessed  immediately  following  training  and  two 
weeks  after  returning  to  work.  The  training  addressed  three 
critical  stages:  problem  finding,  problem  solving,  and 
solution  implementation.  The  findings  suggest  that  the 
creative  problem  solving  training  resulted  in  significant 
and  measurable  effects  both  directly  following  training  and 
two  weeks  after  returning  to  work. 
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Basadur  et  al,  1S82  (continued) 


Intercuriielations  of  Derived  Scores  Representing  the  Six  Dimensions" 


Preference  for  ideaiicn  in 

Pnicitce  of  ideaiion  in 

Probkm 

Problem 

Problem 

Problem 

ProMcm-rinding 

3  'oblem-Mivini 

findinf 

tolvini 

rnidint 

tolving 

performance 

perfornwnce 

Preferane*  for  xkaiion 
in  problem  (indini 

PreCerence  for  ideaiion 

.}} 

in  problem  solvini 

Praciice  of  ideation 

.03 

60 

in  problem  findini 

PMCtiee  of  ideaiion 

26 

.09 

.W 

in  problem  solvini 

.It 

.39 

.32 

$7 

Probkm-findint  performance 

.30 

.37 

.33 

.49 

.8) 

Prnbkm-solvtni  performance 

04 

.33 

.21 

.33 

.22 

.73 

*  Cronbsch  alphu  have  bc«n  insaned  alone  lh«  diafonal. 
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training  in  CReATlVE  PROBLEM  SOLVING 


Basadur  et  al,  1982  (continued) 


PacKRiNct  roa  lotATioM  in  PaotitM  SoiviMfi  MtAfuaio  tr  QutsriONNAiat 


Meaeufcr' 

Fini  Reaciion 
to  New  Unusual 
Product  Ideas 

Total  Reaction 
to  a  New  Unusual 
tVoduci  Idea 

Oefenal  of 
Cniical 
JudgnMni 

Group  mcaM 

1.  Eapahoienul 

5.5 

6.0 

12.6 

2.  Plaeato 

4.5 

3.1* 

I0.»* 

3.  UniretMd 

4.3* 

4.0 

04'* 

Tens  of  suuiiKal  iifnirieance  (a  «  05) 
Croup  1  vs  2  ra 
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•p  -  .020 
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Croup  1  vs  3 

•p  -  .031 

ftt 

•*p  -  .001 

Group  2  vs  3 

m 

ip  «  .062) 

itt 

f(2.42) 

36* 

3.1* 
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*  All  mcaMiFH  taken  iminctiiawly  after  iraininvplaceboiiWAplacebo. 
•  p  *  .05. 
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PRACTICE  or  Ideation  in  Psoeiem  Finding  as  Measueed  et  TArE-RECOEOEO. 
Thinkinc-out-Lduo"  Tase  oueinc  an  Individual  Diveecent  Thinking  Task 


McE.siirct* 

No.  of  Nepiive  Judinsenis 
Made  per  Minute 

Amount  of  Time  tin  saeondsi 
Spent  in  Negative  EvaluatKMi 

Croup  riKans 

1.  Expcnmenul 

06 

2.1 

2.  Placebo 

.J4*» 

II.O* 

).  Untreated 

16** 

I71** 

Tests  of  suiisiieal 

siEnificance  ta  •  OSI 

Croup  1  vs  2 

**p  -  004 

•p  A  017 

Croup  1  vs  } 

••p  -  Oil 

••p  A  001 

Gro<ip  2  vs  ) 

m 

IB 

f(2.42) 

5.4** 

4S* 

*  All  mcEsms  ukcn  imMEdiEidy  after  irmuuAnpiactbtyiioaplKtbo. 
•p  «  OS. 

••p  *  .01. 


Peacvice  or  Ideation  in  Peoelem  Solving 
Method  or  MEasueement 

Questioanairt  Inicrvicw 

ChEiifc  NoikcO  in  Bcinc  ChEiite  Noticed  in  Behie 


More  Likely  lo  Psuie 

Less  Likely  lo  Jmbp  m  id  Tiy  Unusual  or 


Measures* 

More  Openminded  lo  New 
Ideas  and  Approaches 

By  self  By  others 

More  Openminded  to  New 
Ideas  and  Approaches 

By  self  By  others 

Conclusioos  u  lo  what 

Is  the  Real  ProMem 

By  self  By  others 

Ciealive  Approaches 

10  Solvint  iVoblemt 

By  self  By  others 

Group  means 

1.  Eipcrimeniul 

A  1.6 

+  1.5 

»)0 

+0.61 

+  2,1 

+  .52 

+  1.1 
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2  Placebo 

rl.f 

+  1.0* 
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■r0.)0* 
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+  19 

+0.6 
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♦a»* 
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a0.I7" 

+0.4** 
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+0,00*  • 

Tests  u(  staiiilKEl  signiflcaoce  la 

Croup  1  vs  2  '*p  A  .001 

A  .05) 

•p  A  .045 
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•p  -  050 

••p  <  .001 

nsfp  -  .0671 

nsip  A  .0761 

••p  A  .002 

Croup  1  vs  1 

**p  A  .001 

•p  A  ,022 

••p  <  ,001 

••p  A  007 

••p  <  .001 

••p  -  .003 

*p  A  .026 

••p  A  001 

Croup  2  vs  1 

OE 

na 

na 

ns 

na 

M 

na* 

M 

F(2.42) 

I4.4** 

1.5* 

13  1** 

5  2** 
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5.I** 

1.5* 

I1.5* 

"  All  measures  taken  2  weeks  a/ier  return  to  work. 
•  II  •  OS. 

•'  P  E  ,01. 
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1982  (continued) 


Basadur  et  al, 


rlUlLf  M-I-INOINO  ftWoaMANCt 

Method  of  incMtiitmtm 

Tape-recorded,  ihinking-oui-loud"  u»k:  Wishes  lenenud  for  a  new 

product  of  the  future  Quesitontuire 


AinouM  of  liac 


Measures* 

Creative  quality 
tbased  on  Jackson  k 
Messick's  4  criieru  of 
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number  of 
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m 

m 

m 
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•p  d  .01. 

••p  «  .03. 
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Basadur  et  al,  1982  (continued) 
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Boreham,  N.C.  <1985) .  Transfer  of  training  in  the  generation  of 
diagnostic  hypotheses:  The  effect  of  lowering  fidelity  of 
simulation.  British  Journal  of  Educational  Psychology.  55. 
213-223. 


Task:  Simulated  problem  diagnosis 

Taxa;  Problem  solving 

Type  of  Data:  Group 

Subjects:  40  adults 

Design:  Repeated  measures 

Training: 

1.  Subjects  were  trained  for  either  divergent  or 
convergent  hypothesis  testing  in  problem  solving. 

Independent  Variable (s) : 

1 .  Hypothesis  testing  training  for  simulated 
diagnostic  tasks  used  in  salt  packaging  plant 

Dependent  Measure (s): 

1 .  Number  of  different  categories  of  hypotheses 
generated  during  pre-test  and  post-test 

Abstract:  This  study  tested  whether  practice  of  diagnostic 
problem  solving  inhibits  learning  by  forcing  the  learner  to 
practice  the  convergent  type  of  hypothesis  typical  of  most 
diagnostic  situations.  Forty  adults  practiced  on  a 
simulation  of  a  diagnostic  task  and  were  then  tested  on 
their  ability  to  generate  hypotheses  on  a  new  problem.  The 
degree  of  convergence /divergence  of  the  hypothesis  testing 
was  varied  experimentally.  Results  demonstrated  that 
subjects  who  practiced  the  more  convergent  type  of 
hypothesis  testing  acquired  less  ability  than  those  who 
practice  the  more  divergent  type  of  hypothesis  testing. 


Boreham,  1985  (continued) 


Pm*  ano  PosT'TfST  ScMu  roft  na  Nummk  or  CATiootuas 
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Briggs,  G.E.,  &  Naylor,  J.C.  (1965) .  Experiments  on  team 
training  in  a  CIC  environment  (NAVTRADEVCEN  1327-1) . 
Columbus,  Ohio;  Human  Performance  Center  (OTIC  No.  AD- 
608309)  . 


Tas)^:  Simulated  radar-controlled  aerial  intercept 

Taxa:  Information  processing/fine  motor 

Type  of  Data:  Group 

Subjects:  24  Radar  Controllers 

Design:  Factorial,  2  (taslc  complexity)  x  2  (taslc 

organization)  x  2  (replacement  training) 

Training: 

1 .  Subjects  were  trained  on  a  simulated  radar- 
controlled  aerial  intercept  taslc  either  independently 
or  interactively. 

2.  Subjects  were  given  either  a  relatively  complex  or 
a  relatively  simple  task. 

3.  Subjects  were  either  given  additional  training 
sessions  or  fewer  training  sessions. 

Independent  Variable (s): 

1.  Interaction  of  team  members  in  training  for  radar 
control  task 

2.  Complexity  of  radar  control  task 

3.  Length  training  for  radar  control 

Dependent  Measure (s) : 

1.  Efficiency  score  composed  of  number  of  successful 
intercepts,  time,  and  fuel  consumed 

Abstract:  Two  experiments  designed  to  study  team  training 
in  a  Combat  Information  Center  (CIC)  type  environment  were 
conducted.  The  transfer  task  of  both  experiments  required 
the  two  team  members  to  coordinate  their  radar-controlled 
air  intercepts.  Training  task  fidelity  was  varied  in  terms 
of  input  and  output  features  of  the  task  environment  in  a 
first  experiment  and  in  terms  of  task  definition  in  a  second 
experiment.  Both  input  and  output  fidelity  affected  the 
ability  to  coordinate  at  transfer.  Coordination  performance 
at  transfer  tended  to  be  directly  related  to  the  emphasis 
placed  on  coordination  skills  during  training. 
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Briggs  et  al.,  1965  (continued) 


Percentage  of  Subjects  at  Each  college 
Level  for  the  Three  Experiments 


Experiment 

Years  in  School 

1  2  3  4  5 

.  .  _ 

I 

25.7  23.5  23.5  19.1  8.0 

II 

30.0  18.7  15.1  20.5  15.1 

III 

37.5  31.2  20.8  6.2  4.1 

Briggs  et  al.,  1965  (continued) 


Analysis  of  Efficiency  Data  for  Sessions  1  and  2 

of  Experiment  III 


Source 

df 

MS 

F 

Transfer  Organization  (TrO) 

1 

385,193,984 

1.02 

Training  Organization  (TO) 

1 

3,  667,  968 

- 

TO  X  TrO 

1 

728,579,072 

1.92 

Teams  within  groups  (Ts/G) 

20 

378,741,424 

Sessions  (S) 

1 

5,469,124,608 

21.38** 

S  X  TrO 

1 

493,853,184 

1.93 

S  X  TO 

1 

100,352 

- 

S  X  TO  X  TrO 

1 

1  ! 

657,  139,  712 

2.57 

S  X  Ts/G 

20 

255,743,856 

**  p  <  .01 
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Briggs  et  al.,  1965  (continued) 


Transfer 

Task 

I 

Praininc 

j  Task 

Low  Fidelity 
Interact,  Indep. 

Sums 

High 

Interact . 

Fidelit 

Indep. 

Sums 

Interaction 

19, 193 
(8) 

10,922 
(6)  1 

30, 115 

16, 695 
(7) 

9,213 

(5) 

25, 908 

Independent 

7,872 

(4) 

11,175  i 
(2)  ! 

19,  047 

11, 630 
(3) 

9,975 

(1) 

21, 625 

Training  Task 
Fidelity 


Main  Effects  (Sums) 


Tas)c  Organization 


Training  Task 
Interact.  Indep. 


Low  High  ]  Interact.  Indep. 


49,162  47,733  55,410  41,275 


Transfer  Task 
Interact,  Indep. 


56,023 


40,672 


Briggs  et  al.^  1965  (continued) 


Analyses  of  the  Efficiency  Data  of  Experiment  II 


Source 

df 

Sessions  1 
MS 

&  2 

Sessions  3 
MS 

&  4 

F 

F 

Fidelity  (F) 

1 

4, 642, 176 

- 

2,329, 088 

- 

Training 

Organization  (TO) 

1 

349, 123,456 

4.23* 

697, 792 

* 

Transfer 

Organization  (TrO) 

1 

412, 416, 334 

5.00* 

78,733, 184 

2.18 

F  X  TO 

1 

30,711, 936 

- 

9,247, 808 

- 

F  X  TrO 

1 

80,538,240 

- 

60,456, 000 

1.67 

TO  X  TrO 

1 

528,721,792 

6.41* 

64,082, 752 

1.77 

F  X  TO  TrO 

1 

58,196,864 

- 

3,849, 152 

- 

Teams  within 

Groups  (Ts/G) 

48 

82,470,515 

36, 175, 124 

Sessions  (S) 

1 

684, 934,400 

34.84** 

26,712, 832 

2.52 

S  X  F 

1 

684,544 

- 

589, 696 

- 

S  X  TO 

1 

95,744 

- 

12,496, 368 

1.19 

S  X  TrO 

1 

54,  917,  888 

2.79 

6,565, 824 

- 

S  X  F  X  TO 

1 

7,382,016 

- 

19,252, 056 

1.82 

S  X  F  X  TrO 

1 

91,826,  432 

4 . 67* 

5,068, 160 

- 

S  X  TO  X  TrO 

1 

27, 614,208 

1.40 

9,108, 672 

- 

S  X  F  X  TO  X  TrO 

1 

21, 376 

- 

16,354,240 

1.54 

S  X  Ts/G 

48 

19, 660, 482 

10, 605, 585 

*  p  <  .05 

**  p  <  ,01 
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Briggs  et  al.,  1965  (continued) 


The  Interaction  of  Sessions  and  Replacement 
Training  Variables  in  Experiment  I 


Amount  of 

Replacement 

Transfer 

Session 

Training 

2 

3 

High 

19, 359 

10, 020 

Low 

11, 826 

12, 147 
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Briggs  et  al.,  1965  (continued) 


Analyses  of  the  Efficiency  Data  (Fuel  Consumed  per  Successful 

Interception)  of  Experiment  I 


Source 

df 

Sess.  1&2 

F  (MS) 

Sess.  2&3 

F  (MS) 

Sess.  3&4 
F  (MS) 

Task  Organization 

. 

3.64 

7.66* 

(0) 

1 

(7, 611) 

(275, 134) 

(140,764) 

Replacement  Training 

2.68 

1.54 

- 

(R) 

1 

(1,382,521) 

(116, 894) 

(2,705) 

Task  Complexity  (C) 

1 

12.77** 

10.99** 

19.74** 

(6,577,565) 

(829, 872) 

(362,400) 

0  X  R 

1 

- 

- 

- 

(403,361) 

(3,894) 

(7,998) 

0  X  C 

1 

- 

~ 

- 

(19,066) 

(5,  838) 

(7,  998) 

R  X  C 

1 

- 

- 

- 

(86,  754) 

(5,598) 

(1.023) 

0  X  R  X  C 

1 

- 

- 

- 

(87, 643) 

(566) 

(1,158) 

Teams  withing  Groups 

- 

- 

(Ts/G) 

24 

(514,764) 

(75,  490) 

(18, 356) 

Sessions  (S) 

1 

21.59** 

5.31* 

4.38* 

(11,224,0691 

(325, 329) 

(81, 647) 

S  X  0 

1 

- 

- 

- 

(337,544) 

(30,238) 

(602) 

S  X  R 

1 

- 

6.09* 

2.53 

(49,  669) 

(373,372) 

(47,145) 

S  X  C 

1 

3.24 

2.04 

- 

(1,688, 088) 

(125, 648) 

(2,069) 

S  X  0  X  C 

1 

- 

- 

(223,235) 

(5,575) 

(2,323) 

S  X  0  X  R 

1 

- 

1.08 

- 

(419,085) 

(66, 670) 

(8,531) 

S  X  R  X  C 

1 

- 

- 

— 

(81, 018) 

(4,214) 

(11,610) 

S  X  0  X  R  X  C 

1 

- 

- 

- 

(65, 620) 

(258) 

(1,742) 

S  X  Ts/G 

24 

(519,763) 

(61,295) 

1 

(18,611) 

*  p  <  .1)5 
**  p  <  .01 
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FUEL  CONSUMED  PER  INTERCEPT  (Ibt/hit) 


SESSIONS 


Briggs  et  al.,  1965  (continued) 
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FUEL  CONSUMED  PER  INTERCEPT  (lb«/hit) 


Briggs  et:  al.,  1965  (continued) 


FUEL  CONSUMED  PER  INTERCEPT  (Ibs/hit) 


Briggs  et  al.,  1965  (continued) 


Eirielaney  data  for  iha  Intaraetlon  organization  o<  tht  oporatlonal  taak.  Exptrimant  III. 
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FUEL  CONSUMEO  PER  INTERCEPT  (Ibt/hit) 


Briggs  et  al.,  1965  (continued) 


2S,000{ 


20,000 


Tran»t«r  Task 
Organization 


Training  Task 
FIdallty 


1  Indapandsnt  High 

2  Indapandani  Low 

3  Intaraetion  High 

4  Intaraetion  Low 


SESSIONS 

Elllolaneir  data  for  tha  Indopandant  organization  of  tha  oparntlonal  task.  Exparlmant  II. 
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FUEL  CONSUMED  PER  INTERCEPT 
(loq^  pounda/hli) 


Briggs  et  al.,  1965  (continued) 


sessions 


Sy<t«m  •fflelanoy  for  Eiporimoni  I. 
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Card,  S.K.,  English,  W.K.,  and  Burr,  B.J.  (1978).  Evaluation  of 
mouse,  rate-controlled  isometric  joystick,  step  keys,  and 
text  keys  for  text  selection  on  a  CRT.  Ergonomics.  21.  601- 
613. 


Task:  Text  selection  on  a  CRT  display 
Taxa:  Fine  motor/information  processing 
Type  of  Data:  Group  means 
Subjects :  5  undergraduate  students 

Design:  Repeated  measures 
Training: 

1 .  Subjects  were  trained  on  four  different  text 
selecting  devices 

2.  Performance  was  tested  during  latter  section  of 
training  trials 

Independent  Variable (s): 

1.  Distance  from  starting  point  to  Positioning  device 
(mouse,  joystick,  step  keys,  and  text  keys) 

Dependent  Measure (s) : 

1.  Positioning  speed 

2.  Homing  time 

Abstract :  This  study  evaluated  four  devices  (mouse, 
isometric  joystick,  step  keys,  and  text  keys)  with  respect 
to  the  speed  with  which  they  could  be  used  to  select  text  on 
a  CRT  display.  Results  demonstrated  that  the  mouse  was 
fastest  and  produced  the  least  errors.  It  was  shown  that 
variations  in  positioning  time  with  the  mouse  and  joystick 
are  accounted  for  by  Fitts's  Law.  With  regard  to  the  mouse, 
the  measured  Fitts's  Law  slope  constant  is  close  to  that 
found  in  other  eye-hand  tasks  leading  to  the  conclusion  that 
positioning  time  with  the  mouse  is  almost  the  minimal 
achievable.  With  regard  to  key  devices,  positioning  time 
was  shown  to  be  proportional  to  the  number  of  keystrokes 
that  must  be  typed. 
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Card  et  al,  1978  (continued) 


Summaiy  of  Models  for  Positioning  Time  (T^ 


Device 

Model  (times  in  s) 

Mouse 

=  1.03  +  0.096  log,  (0/S  +  0.5) 

0.07 

0.83 

Joystick 

«  0.99  +  0.220  log,  (D/S  +  0.5)* 

0.13 

0.89 

log,  (O/S  +  0.5)'* 

0.07 

Susp  Keys 

-  1.20  +  0.052  (OyS,  +  0/5/ 

0.54 

0.84 

*  0.98  +  0.074  (PJS,  +  O/Sj* 

0.18 

0.95 

Text  Keys 

«  0.66  +  0.209 

0-24 

0-89 

‘Least  squares  fit  to  all  data  points. 

^it  for  number  of  keystrokes  DJS,  +  <40. 

1.0831  squares  fit  to  all  data  points. 

l^ltting  a  separate  line  with  slope  0.1  bits  for  each  distance 
where  HOME  key  unlikely  to  be  used. 
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Card  et  ai,  1978  (continued) 


Positioning  time  for  key  devices  as  a  function  of  predicted  number  of  keystrokes. 
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Card  et  al/  1978  (continued) 
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Card  et  al,  1978  (continued) 


Overall  Times 

Movemeni  lime  for  non-error  trials  is)  Error  rate 

OcMce  Homing  Time  Positioning  Time  Total  Time 


M 

SD 

M 

SD 

M 

SD 

M 

so 

Mouse 

0-36 

0-13 

t  29 

0-12 

166 

048 

•  • 

Jovstick 

026 

Oil 

1  57 

054 

1  83 

057 

11*. 

31% 

Step  iCeys 

O':! 

0)0 

2  31 

152 

251 

164 

ij*. 

3J*. 

Test 

OJ2 

061 

195 

1  30 

226 

1  70 

28% 

I 

I 


Card  et  al,  1978  (continued) 


Learning  Curve  Parameten 


DEVICE 

T, 

X 

Learning  Curve 

s. 

Jt^ 

(s) 

Equation* 

(s) 

Mouse 

2.20 

0.13 

r^  =  2.20iV'"" 

0.12 

0.66 

Joystick 

2.19 

0.08 

r„«2.19^**^" 

0.08 

0.62 

Stq>Keys 

3.03 

0.07 

rw“3.03Af"” 

0.11 

0,39 

Text  Keys 

3.86 

0.15 

0.16 

0.61 

*N  is  number  of  trial  blocks.  There  are  20  trials  in  each  block. 
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POSITIONilM:  TIME  (il  POSITIOMING  TIME  l<) 


Card  et  a.l,  1978  (continued) 
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Cockrell,  J.T.  (1979)  .  Effective  training  for  target 

identification  under  degraded  conditions  (ARI  Technical 
Paper  358) .  Alexandria,  VA:  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences. 


Task:  Visual  discrimination 

Taxa:  Visual 

Type  of  Data:  Group 

Subjects:  96  subjects  who  had  just  completed  basic 

training  or  who  were  in  advanced  individual  training 

Design:  Repeated  measures 

Training: 

1.  Subjects  were  trained  on  targets  of  differing 
visibility 

Independent  Variable (s) : 

1.  Target  quality  during  training  (degraded  vs. 
normal) 

Dependent  Measure (s): 

1.  Mean  number  of  correct  identifications  of  degraded 
targets 

Abstract :  This  study  explored  the  relevance  of  the 
psychological  concept  of  overshadowing  to  military  target 
identification  training.  The  concept  would  predict  that  the 
most  outstanding  features  of  the  target  will  capture  the 
trainees  attention  more  than  any  of  the  other  features. 

This  suggests  that  soldiers  will  have  difficulty  identifying 
degraded  targets  unless  they  are  trained  to  do  so.  This 
experiment  forced  trainees  to  focus  attention  on  increasing 
numbers  of  features  by  occluding  the  outstanding  features. 
Results  demonstrated  the  relevance  of  overshadowing  and 
suggested  that  most  training  should  concentrate  on  degraded 
views . 
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2 

67% 


3 

33% 


Tests 


making  perfect  score  on  both 
experiment  test  and  retest. 
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Mean  Number 


o 

3 

2 

1 


3 

33% 


Mean  number  of  correct  responses  on  main 
experiment  tests. 


1  2 

100%  67% 

Tests 
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Percent  of  Subjects  Making  Perfect  Score 


Percent  of  Subjects  Making  Perfect  Score 


Mean  Number  of  Correct  Responses 


Mean  Number  of  Correct  Responses 


Cockrell/  1979  (continued) 


J _ I _ 1  _  t.  ..  1  1 

1  2  3  4  1  2 

Training  Trials  Tests 


1 
1 

Retests 


Mean  number  of  correct  responses  for  each  group  during 
the  warm-up  phase,  (Smoothed  curve.) 


Eberts,  R,E.  (1987) .  Internal  models,  tracking  strategies,  and 
dual-task  performance.  Human  Factors.  29,  407-419. 


Task;  Visual  tracking  and  auditory  detection  task 
Taxa:  Visual/Information  processing 
Type  of  Data:  Group 
Subjects:  24 

Design:  3  (parabola  augmentation,  point  augmentation,  or  no 

augmentation)  x  2  (single  or  dual  task) 

Training; 

1.  Subjects  performed  either  a  single  task  (auditory 
or  visual)  or  a  dual-task  (both  auditory  and  visual) . 

2.  Subjects  were  given  either  point  cues,  parabola 
cues,  or  no  cues. 

Independent  variable (s); 

1.  Task  type  (single  or  dual) 

2.  Augmentation  type  (point  or  parabola  cues) 
Dependent  Measure (s); 

1.  Root  mean  squared  error  (RMS)  tracking  performance 

Abstract:  This  study  investigated  the  effects  of  internal 
models  and  tracking  strategies  on  workload.  Internal  models 
and  tracking  strategies  were  varied  by  providing  subjects 
with  augmenting  cues.  Performance  of  two  groups  provided 
with  different  types  of  display  augmentation  (parabola  or 
point  cues)  was  compared  with  that  of  a  control  group  for 
single-task  tracking  training.  Results  indicated  that  both 
point  and  parabola  augmentation  reduced  workload  when 
displayed  on  cued  trials.  On  the  non-cued  trials,  the 
parabola  augmentation  reduced  workload  more  than  the  point 
augmentation.  This  suggests  that  a  consistent  tracking 
strategy  or  the  development  of  a  visually  based  internal 
model  is  needed  to  lessen  workload. 
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EbertS/  1987  (continued) 


SINGLE  DUAL 

TASK  TYPE 


RMS  Tracking  error  as  a  function  of  task 
type  (Single  or  dual)  for  the  cued  and  noncued  trials 
of  the  two  augmented  groups  in  Experiment  2. 
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Edwards,  B.J.,  Weyer,  D.C.,  &  Smith,  B.A.  (1979).  Undergraduate 
pilot  training:  Visual  discrimination  pre-training  for 
landing  task  (AFHRL-TR-78-78) .  Brooks  Air  Force  Base, 
Texas:  Air  Force  Systems  Command  (OTIC  No.  AD-A068141) . 


Task:  Visual  discrimination 

Taxa:  Visual 

Type  of  Data:  Group 

Subjects:  38  males  students  at  Williams  AFB 
Design:  Repeated  measures 
Training : 

1.  Subjects  received  either  cognitive  pre-training, 
Visual  Discrimination  Pre-training  (VDPT) ,  VDPT  and 
Advance  Simulator  for  Pilot  Training  (ASPT) ,  only  ASPT, 
or  no  pre-training. 

Independent  Variable(s): 

1 .  Pre-training  (multimedia  or  no  pre-training) 
Dependent  Measure (s) : 

1.  Instructor-rated  transfer  performance  in  the  T-37 
aircraft 

Abstract :  This  study  examined  the  utility  of  training  task¬ 
relevant  visual  discrimination  stalls  as  prerequisite 
behaviors  for  subsequently  taught  landing  skills.  A  multi- 
media  training  package  was  developed  and  used  to  teach 
visual  skills  to  student  pilots  prior  to  their  training  for 
landing  procedures  in  the  T-37  aircraft.  Transfer  of  visual 
discrimination  skills  of  the  landing  field  environment  was 
assessed  by  measuring  students'  landing  skills  in  the 
Advanced  Simulator  for  Pilot  Training  and  in  the  T-37 
aircraft .  Results  indicated  that  pre-trained  and  non-pre- 
trained  performance  was  not  significantly  different. 
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Edwards,  1979  (continued) 


Overall  Perfermanc*  Rating 

by  IP  (or  Final  Turn  Trials  in  Aircraft 

Group  1 

Group  2 

Group  3 

Group  4 

Group  5 

Control 

Procedures 

VOPT 

ASPT 

VOPT/ASPT 

Only 

17 

19 

24 

14 

32.5 

12 

8 

36 

2 

9 

34.5 

10 

5 

30 

21 

3,5 

3.5 

15.5 

19 

37 

22.5 

1 

25 

22.5 

19 

13 

15.5 

6.S 

11 

26.5 

32,5 

6.5 

28 

34.5 

38 

30 

30 

26.5 

R^-134.5 

R^-161.S 

R^-125.5 

R^«156.5 

R^»182 

H-1.88  (p  <.75) 
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Fisk/  A.D.,  &  Lloyd/  S.J,  (1988).  The  role  of  stimulus-to-rule 
consistency  in  learning  rapid  application  of  spatial  rules. 
Human  Factors.  30.  35-49. 


Task:  Decision  making 
Taxa:  Problem  solving 
Type  of  Data:  Group 
Subjects:  6  undergraduates 

Design:  Repeated  measures 
Training: 

1.  Subjects  repeated  a  decision-making  task  over  6048 
trials . 

Independent  Variable (s): 

1 .  Memory  set  size 

2.  Possibility  of  correct  decision  (positive  or 
negative  trials) 

Dependent  Measure (s): 

1.  Reaction  time  in  dual-task  situation 

Abstract:  This  study  addressed  the  effects  of 
intercomponent  consistency  on  skill  acquisition  in  a  class 
of  cognitively  demanding  tasks  that  required  rapid 
integration  of  information  and  rapid  rule  application. 

After  extensive  practice/  subjects'  performance  was 
remarkably  similar  to  performance  observed  in  traditional 
perceptual  learning  tasks.  This  suggests  that  mechanisms 
underlying  traditional  learning  in  visual  search  and  rule- 
based  spatial  learning  are  similar.  Subjects  who  were 
trained  so  that  consistent  stimulus  to  rule  association  were 
built  up  and  strengthened  with  practice  performed 
qualitatively  and  quantitatively  better  than  those  who  were 
trained  with  inconsistent  stimulus  to  rule  associations. 
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Fisk,  et  al.,  1988  (continued) 


«  I  1  4  %  4  f  •  •  fO  11  It  tJ  14  *4  W  !•  It  M  I*  fl  l« 

GROUPS  Of  SLOCKS 

Average  comparison  slope  estimates  (in 
ms)  for  positive  trials  (solid  circles)  and  negative 
trials  (open  circles).  The  data  are  shown  as  a  f  unction 
of  practice  (groups  of  blocks).  Data  from  Experi¬ 
ment  JA. 


1988  (continued) 


Fisk,  et  al.. 
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AtACTlON  IIME  (n>MC| 


Fisk/  et  al . ,  1988  (continued) 


’  Reaction  time  to  indicate  correctly  the  cor¬ 
rect  capturing  "game-piece.'’  The  data  are  plotted  as 
a  function  of  practice  (groups  of  blocksh  Data  from 
Experiment  /. 
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Johnson,  S.L.  (1978)  Retention  and  transfer  of  training  on  a 
procedural  task;  interaction  of  training  strategy  and 
cognitive  style  (AFOSR-TR-78~1161) .  Bolling  AFB.  DC:  Air 
Force  Office  of  Scientific  Research. 


Task;  Set  up  a  conveyor  line 
Taxa:  Information  processing/Motor 

Type  of  Data:  Group 

Subjects:  60  (36  male,  24  female)  paid  subjects 

Design:  Repeated  measures  (initial  training,  retraining, 
and  transfer) 

Training: 

1 .  Subjects  were  trained  under  one  of  three 
conditions:  conventional,  reproduction,  or  blind. 

Independent  Variable(s): 

1.  Initial  training  strategy  for  "checklist" 
sequential  task:  conventional,  reproduction,  or  blind 
strategy 

Dependent  Measure (s): 

1 .  Sequence  and  setting  errors  on  sequential  task 

2.  Time  to  perform  the  last  trial  on  sequential  task 

3.  Transfer  errors  on  actual  sequential  task  (e.g., 
control  panel  equipment) 

Abstract:  This  study  examined  the  effectiveness  of  three 
different  training  strategies  related  to  initial  training, 
retention,  and  transfer  of  training.  Also,  the  possibility 
of  matching  the  instructional  strategy  and  the  trainees 
cognitive  style  was  evaluated.  The  task  used  was 
representative  of  the  many  sequential  procedures  performed 
which  range  from  operating  master  control  panels  in 
industrial  plants  to  emergency  procedures  in  air  vehicles. 
Results  indicated  that  a)  vividness  of  imagery  interacts 
with  training  strategy,  b)  training  devices  do  not  need  high 
fidelity  to  be  effective  in  training  procedural  tasks,  and 
c)  the  use  of  training  strategy  that  requires  self-cuing  and 
self-feedback  is  effective  in  increasing  the  retention  of 
procedure-follow  skills  independent  of  cognitive  style. 
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NUMBER  OF  ERRORS 


Johnson,  S,L.  (1978) 


LEARNING  CURVES  FOR  THE  THREE  TRAINING  STRATEGIES 
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Kanfer,  R.,  and  Ackerman,  P.  L.  (1989).  Motivation  and  cognitive 
abilities:  An  integrative/aptitude-treatment  interaction 
approach  to  skill  acquisition.  Journal  of  Applied 
Psychology  Monograph.  74 ,  657-690. 


Task:  Air  traffic  controller  task  (ATC) 

Taxa:  Perceptual;  Fine  motor;  Information  processing/ 
Problem  solving 

Type  of  Data:  Group 

Subjects:  552  U.S.  Air  Force  recruits 

Design:  Repeated  measures 

Training: 

1 .  Part-task  training  on  the  ATC  using  either  a 
procedural  or  declarative  strategy 

2.  Subjects  were  given  "goal"  or  "no-goal" 
instructions 

3.  Final  Test  -  6  Trials  on  the  full  ATC  task 
Independent  Variable (s): 

1.  ATC  Part-task  training  strategy:  declarative  or 
procedural 

2.  Goal  setting  strategy:  prior  to  attempting  the 

full  ATC  task,  subjects  were  either  given  "goal"  or 
"no-goal"  instructions 

3 .  Individual  ability  as  measured  by  ASVAB  test 
Dependent  Measure (s) : 

1.  Number  of  planes  landed  on  six  trials  of  full  ATC 

Abstract:  This  study  simultaneously  considered  individual 
differences  in  cognitive  abilities,  self-regulatory 
processes  of  mgtivation,  and  information  processing  demands. 
Evidence  for  this  framework  is  provided  in  the  context  of 
skill  acquisition  in  which  information  processing  and 
ability  demands  change  as  a  function  of  practice,  training 
paradigm,  and  timing  of  goal  setting.  Three  field  based 
experiments  were  conducted  with  1,010  U.S.  Air  Force 
trainees,  in  Experiment  2,  the  basic  ability-performance 
parameters  of  air  traffic  controller  task  and  goal  setting 
effects  late  in  practice  were  examined.  The  results  have 
implications  for  notions  of  ability-motivation  interactions 
and  design  of  training  and  motivational  programs. 
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Plan**  Landed  PlanM  Landed 


Kanfer  et  al.  1989  (continued) 


Planes  landed  at  a  function  of  full  air  traffic  controlkr  task 
trial,  by  condition  and  by  ability  group  (median  split).  (No-goal  condi¬ 
tions  are  in  solid  lines;  g^  conditions  are  in  dotted  lines.  High-ability 
group  it  in  circles;  low-ability  group  is  in  squares.) 


A-50 


Kanfer  et  al.  1989  (continued) 


Trial  Trial 


Ability  ptrforwiCT  corrtiMioniby  ability  factoc  condhicn,  mi  fiillmKAi;  practia  trial.  (Oca- 
erd  ability' i)crforRiaiioe  corrdatioM  are  in  loHd  Gaea  and  filled  aoavrea.  fterteptua:  ipaed'-isTftnnenc* 
corrHatiom  are  in  daabed  Unta  ami  flptn  dielea.) 


A-51 


Kanfer  et  al.  1989  (continued) 


Planes  landed  as  a  function  of  air  traffic  controller  task  trial 
for  final  three  trials,  by  condition  and  by  ability  group  (median  split). 
(No^oal  condition  is  in  solid  lines;  early>goal  condition  is  in  dashed 
lines.  High-ability  group  is  in  circles;  low-ability  group  is  in  squares.) 
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Kanfer  et  al.  1989  (continued) 


Lintern,  G.,  Roscoe,  N.R.,  Koonce,  J.M.,  &  Segal,  L.D.  (1990). 
Transfer  of  landing  skills  in  beginning  flight  training. 
Human  Factors.  32.  319-327. 


Task;  Aircraft  landing  task 
Taxa:  Motor/Visual 
Type  of  Data:  Group 

Subjects:  42  flight  students  with  no  previous  flight 

experience 

Design;  Repeated  measures 
Training: 

1.  Subjects  either  received  traditional  instruction  or 
traditional  instruction  and  supplemental  simultitor 
training. 

Independent  Variable (s): 

1.  Landing  instruction:  supplemental  simulator 
training  or  traditional  training 

Dependent  Measure (s); 

1 .  Raw  tracking  error  on  landing  task 

Abstract:  Beginning  flight  students  were  given  two  sessions 
of  landing  practice  in  a  simulator  with  computer  animated 
contact  landing  display  before  commencing  landing  practice 
in  the  aircraft.  Subjects  in  the  experimental  condition 
received  supplemental  simulator  landing  training  while 
subjects  were  not  afforded  this  treatment.  Results 
demonstrated  that  those  students  in  the  experimental  group 
required  significantly  fewer  pre-solo  landings  in  the 
airplane  than  did  those  in  the  control  group.  Also,  some 
students  were  provided  adaptive  visual  augmentation  during 
simulator  training  which  produced  incremental  transfer 
effects  attributable  to  this  feature. 
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Log  RMS  Lateral  Error 


Lintern  et  al .  1990  (continued) 


Lineup  (left)  and  glideslope  (right)  performance  in  the  simulator  for  the  initial  approach  to  landing. 
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Malloy,  T.E.,  Mitchell,  C.,  &  Gordon,  O.E.  (1987).  Training 

cognitive  strategies  underlying  intelligent  problem  solving. 
Perception  and  Motor  Skills.  64,  1039-1046. 


Taslc:  Problem  solving  task 

Taxa;  Problem  solving 
Type  of  Data;  Group 

Subjects;  54  introductory  psychology  students 

Design:  Repeated  measures 

Training: 

1.  Subjects  either  received  different  degrees  of 
problem  solving  strategy  training  or  received  no 
special  treatment. 

Independent  Variable (s) : 

1.  Cognitive  strategy  training:  using  either  Raven's 
Progressive  Matrices  Test,  brief  exposure  to  the  test, 
or  no  exposure 

Dependenc  Measure (s); 

1.  Problem  solving  performance  on  17  matrix  puzzles 

Abstract ;  This  study  examined  the  extent  to  which  training 
problem  solving  strategies  affects  performance.  Cognitive 
strategies  underlying  excellent  performance  of  intelligent 
people  on  the  Raven' s  Progressive  Matrices  Test  were  used  to 
teach  subjects  in  the  experimental  conditions.  Subjects  in 
the  control  group  were  not  exposed  to  the  training  package 
at  all.  Results  showed  that  from  pretest  to  post-test,  the 
group  who  received  problem  solving  training  performed 
significantly  better  than  those  who  were  not  trained. 
Secondly,  those  who  received  problem  solving  training 
performed  better  than  those  who  were  not  trained  on  a 
Piagetian  task  suggesting  a  broad  improvement  in  cognitive 
functioning. 
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MEAN  NUMBER 
CORRECT  OUT 
OF  14 


Malloy  et  al.,  1987  (continued) 


13-- 

12-- 

11 

10-- 

9-- 

8-- 


+ 


ConUol 

Exposure 

Cognitive  Strategy 

+ 


PRE-TEST 


POST-TEST 


Mean  number  of  correct  solutions  out  of  a  possible  14 
as  a  function  of  type  of  training  and  test 
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Masson,  M.E.  (1986) .  Identification  of  typographically 

transformed  words:  Instance-based  skill  acquisition. 
Journal  of  Experimental  Psychology:  Learning.  Memory,  and 
Cognition.  12.  479-488. 


Task :  Word  search 

Taxa:  Visual/problem  solving 

Type  of  Data:  Group 

Subjects:  24  university  students 

Design:  Repeated  measures 

Training: 

1.  Students  were  presented  a  randomly  ordered  list  of 
24  word  triplets  and  were  then  asked  to  identify 
individual  words .  A  test  phase  ensued  inunediately 
after  training  and  a  similar  procedure  was  repeated. 

Independent  Variable (s): 

1 .  Extent  of  exposure  to  word  triplets 

Dependent  Measure (s) : 

1.  Word  identification  score 

Abstract:  This  study  demonstrated  that  the  transfer  of  word 
identification  skill  was  highly  specific  and  occurs  only 
when  training  and  test  instances  shared  common  letters 
printed  in  the  same  case.  Also,  transfer  of  skill  depended 
on  the  visual  patterns  formed  by  adjacent  letters  and  word 
shape.  Presentation  of  a  word  in  training  and  test  phases 
significantly  improved  test  phase  identification  even  if  a 
unique  visual  pattern  was  used.  The  results  are  consistent 
with  an  instance-based  view  of  word  identification  skill. 
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MEAN  READING  TIME  IN  SECONDS 


Miller,  E.E.  (1975) .  Instructional  strategies  using  low-cost 
simulation  for  electronic  maintenance  (HumRRO  Technical 
Report  HumRRo-FR-WD (TX) -75-20) .  Alexandria,  VA;  Human 
Resource  Research  Organization. 


Task:  Electronic  maintenance  and  problem  solving  on  a 
complex  system  (Improved  HAWK) 

Tcixa:  Problem  solving 

Type  of  Data:  Group 

Subjects:  78  students  enrolled  in  the  Improved  HAWK  Firing 

Section  Mechanics  Program  of  Instruction 

Design:  Between  subjects,  no  repeated  measures 

Training: 

1 .  Subjects  either  received  traditional  instruction  or 
traditional  instruction  and  supplemental  training. 

Independent  Variable (s): 

1 .  Instructional  training  for  Improved  Hawk 

Dependent  Measure (s): 

1 .  Practical  quiz  test  score  for  Improved  Hawk 

2 .  Score  on  standardized  quiz  for  Improved  Hawk 

Abstract:  This  study  examined  a  variety  of  instructional 

strategies  that  employ  visual  aids  to  improve  electronic 
maintenance  training  on  a  complex  system  (Improved  Hawk) . 
Large  color  photos  of  control/indicator  panels,  supplemented 
by  color  slides,  were  developed  specifically  for  use  in  the 
classroom.  Supplementary  exercises  were  developed  to  be 
used  by  students  while  awaiting  their  participation  on  the 
equipment.  Results  indicated  that  subjects  receiving 
experimental  training  performed  significantly  better  with 
respect  to  checks,  adjustments,  and  fault  isolation  compared 
with  the  control  group. 
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Miller,  1975  (continued) 


PERFORMANCE  ON  FAULT  ISOLATION  PROBLEMS 
IN  SECOND  EXAMINATION  PERIOD 


Problem  1  (Crystal  Detector) 


Experimental 

DIO  NOT 
COMPLETE 

IN  TIME 

COMPLETED 

IN  TIME 

ERRORS 

1 

2  or  more 

3 

35 

B 

12 

19 

Control 

11 

28 

5 

29 

•  5.3,  p  <  .05  x^2  “  5.3,  n.s. 


Miller,  1975  (continued) 


PERFORMANCE  ON  FAULT  ISOLATION  PROBLEMS 
IN  FIRST  EXAMINATION  PERIOD 


Problem  1  (Interlock) 


Experimental 

DIO  NOT 
COMPLETE 

IN  TIME 

COMPLETED 

IN  TIME 

ERRORS  1 

0 

1 

2  or  more  ! 

5 

34 

27 

10 

1 

2  i 

Control 

19 

20 

mm 

14 

14  i 

x^i  -  13.2,  p  <  .001  xS  *  16^4.  p  <  .001 


Problem  2  (+100  VDC  Power  Supply) 


Experimental 

DID  NOT 
COMPLETE 

IN  TIME 

COMPLETED 

IN  TIME 

ERRORS  ! 

m 

1 

2  or  more  j 

13 

26 

15 

20 

4 

Control 

28 

11 

6 

13 

20 

X^i  •  10.6,  p  <  .001  •  15.9,  p  <  .001 
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Morris,  N.M,,  &  Rouse,  W.B.  (1985).  The  effect  of  type  of 

knowledge  upon  human  problem  solving  in  a  process  control 
task.  IEEE.  1^,  698707. 


Task:  Process  control  task  and  problem  solving 

Taxa:  Problem  solving 

Type  of  Data:  Group 

Subjects:  32  engineering  students 

Design:  2 (Principles)  x  2 (Procedures)  factorial  design  with 
Repeated  measures 

Training: 

1 .  Subjects  either  received  minimal  instructions, 
minimal  instructions  plus  principles,  minimal 
instructions  plus  procedures,  or  all  instructions 
(minimal  instructions,  principles,  and  procedures) 
before  being  tested  in  a  control  task  problem  solving 
task . 

Independent  Variable (s): 

1 ,  Instructional  training  for  dynamic  production 
process  using  simulator  (PLANT) 

Dependent  Measure (s): 

1 .  Production  as  measured  on  dynamic  production 
process  simulator  (PLANT) 

Abstract:  This  study  examined  the  question  of  what  the 
operator  of  a  dynamic  system  needs  to  know.  The  study  used 
PLANT,  a  simulation  of  a  generic  dynamic  production  process. 
Knowledge  of  PLANT  was  manipulated  by  providing  different 
types  of  instructions  to  four  different  groups.  There  was 
no  significant'  difference  between  groups  receiving  different 
instructions.  However,  subjects  who  received  better 
instructions  controlled  the  system  in  a  more  stable  fashion. 
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Morris  et  al 


1985  (continued) 


•  f 


Correlations  Between  Dependent  Measures? 


PROD 

TRIPS 

NOPEN 

var 

FIX 

test2 

SECTl 

sect2 

TRIPS 

-0.437* 

NOPEN 

0.673* 

-0.706* 

var 

-OS  74* 

0.967* 

-0.768* 

FIX 

-0.429* 

0.141 

-0.234 

0.218 

test2 

-0.200 

0.313 

-0.258 

0.107 

SECTl 

-0.021 

0.261 

-0.189 

0.268 

-O.lOO 

0.148 

sect2 

0.190 

-0.238 

0.366* 

-0.292 

0.225 

0.860* 

0.040 

sect3 

0.105 

-0.161 

0.157 

-0.197 

-0.056 

0.661* 

-0.022 

0.225 

‘prod 

TRIPS 

nopen 

VAR 

FIX 

TE5T2 

SECTl. 

SECT2.  SECT3 
SECTl 
SECT2 
sect3 

<  0.05 


average  produciion/iteraiion 

numto  of  automatic  valve  trips/iteration 

avenge  number  of  valves  open/iteration 

variance  of  tank  heights  in  PLANT 

average  time  to  diagnose  valve  and  pump  failures 

overall  score  on  test  2 

scores  (percent  correct)  on  subsections  of  test  2 
minimal  questions 
procedural  questions 
principles  questions. 
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Myers,  G.L.,  &  Fisk,  A.D.  (1987).  Training  consistent  task 
components:  Application  of  automatic  and  controlled 
processing  theory  to  industrial  task  training.  Human 
Factors.  29.  255-268. 


Task:  Telecommunication  processing  task 
Taxa:  Information  processing/Visual 

Type  of  Data:  Group 

Subjects:  8  AT&T  Communications  employees  (five  males, 
three  females) 

Design:  simple  measurement  (independent  variable  was  only 
statistically  manipulated) 

Training: 

1 .  Subjects  were  trained  under  either  the  consistent 
mapping  or  variable  mapping  condition. 

Independent  Variable (s) : 

1.  Mapping  of  information  (variably  or  consistently 
mapped)  in  training  for  a  telecommunications  type  of 
visual  search  task 

Dependent  Measure (s): 

1 .  Speed  and  Accuracy  on  telecommunications  visual 
search  task 

Abstract:  This  study  examined  the  generality  of 
automatic/controlled  processing  training  principles  to  rich, 
complex  tasks.  Subjects'  tasks  were  modelled  after  a  job 
function  performed  in  the  telecommunications  industry.  The 
tasks  required  subjects  to  process  conjunctions  of 
information.  Large  quantitative  and  qualitative  differences 
were  reported  between  consistently  and  variably  mapped 
training  conditions.  Results  indicate  that  theories  of 
automatic  and  controlled  processing  may  be  expanded  to 
include  domains  of  rich,  complex  industrial  tasks. 
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Latency  (msec)  Latency  (msec) 


Myers  et  al./  1987  (continued) 


Response  latency  as  a  function  of  practice  in  Experiment  2.  The  solid 
lines  with  no  circles  or  squares  represent  previous  VM  or  CM 
perfomiance  levels.  CM  (open  circles)  =  consistent  attending; 

VM  (filled  squares)  =  inconsistent  attending. 


Response  latency  as  a  function  of  serial  position  for  both  CM  (open 
circles)  and  VM  (filled  squares)  in  Experiment  2.  (CM  =  consistent 
attending;  VM  =  inconsistent  attending.) 
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Latency  (msec)  Latency  (msec) 


Myers  et  al.,  1987  (continued) 


Response  latency  as  a  function  of  serial  position  for  both 
CM  (open  circles)  and  VM  (filled  squares).  The  data  are 
from  the  last  group  of  blocks. 


Response  latency  as  a  function  of  the  target's  location  on  the  display 
for  l»th  CM  (open  circles)  and  VM  (filled  squares).  Target 
Location  1  corresponds  to  the  upper-left*most  display  position. 
Target  Location  6  represents  the  lower-right-most  display  position. 
The  data  are  from  the  last  group  of  blocks 
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Patrick,  J.,  and  Haines,  B.  (1988).  Training  and  transfer  of 
fault-finding  skill.  Ergonomics.  31(2) .  193-210. 


Task;  Fault  finding  task 

Taxa;  Problem  solving/information  processing 

Type  of  Data:  Group 

Subjects;  24  University  students 

Design:  Repeated  measures 

Training; 

1 .  Technical  story  or  set  of  diagnostic  heuristics 

2.  Final  Test  -  "Transfer"  fault  finding  test  using  a 
different  scenario 

Independent  Variable (s); 

1.  Training  materials  (technical  story  vs  heuristics) 
Dependent  Measure (s) : 

1 .  Accuracy  and  Speed  of  fault  diagnosis  test  with 
different  scenario  on  transfer  test  scenario 

Abstract;  This  study  investigated  the  effect  of  different 
training  materials  on  training  and  transfer  of  fault-finding 
skill.  Two  simulated  chemical  plants  were  used  as  the 
fault-finding  domain.  The  training  materials  consisted  of 
either  a  technical  story  or  diagnostic  heuristics.  Both 
conditions  improved  at  fault-finding  during  training. 
However,  the  technical  story  condition  had  better  overall 
performance  during  training.  Transfer  was  high  and  positive 
for  both  conditions.  Results  are  discussed  with  respect  to 
the  role  of  theory  in  training  fault-finding. 
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Patrick  et  al.,  1988 


Accuracy  and  speed  (nuns)  of  diagnoses  dunng  fault-linding  tests  for  the  experimental 

conditions. 


Fault-finding  tests 

Test  1 

Test  2 

‘Old- 

faults 

Test  3 

‘New  (same  category)' 
faults 

Condition  T(FP2) 

Accuracy 

mean 

0^10 

020 

053 

0-43 

s.d. 

0-11 

019 

030 

020 

Speed 

mean 

399 

3<)9 

2-22 

2-87 

s.d. 

0-8S 

077 

067 

093 

Condition  H(FP2) 

Accuracy 

mean 

0-11 

024 

053 

023 

s.d. 

013 

018 

021 

017 

Speed 

mean 

401 

2-70 

205 

2-76 

s.d. 

083 

089 

052 

055 

Condition  H(CDl) 

Accuracy 

mean 

022 

022 

038 

030 

s.d. 

018 

021 

027 

019 

Speed 

mean 

419 

296 

2-57 

2-97 

s.d. 

049 

067 

058 

041 
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Patrick  et  al.,  1988 


Transfer  of  training  from  FP2  to  CD{  plant  in  terms  of  accuracy,  speed  and  number  o 

instrument  readings. 


Training/test  conditions 

T(FP2)-Tcst  4 

H(FP2)-Tcst  4 

H(CDlf-Tcst  1 
(control) 

Accuracy 

mean 

0-68 

060 

022 

s.d. 

a20 

022 

018 

Speed  (min) 

mean 

2-12 

2-23 

4  19 

s.d. 

0-51 

072 

049 

Instrument  readings 

mean 

6-08 

5-88 

1247 

s.d. 

I-5I 

1  89 

608 
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Patrick  et  al,#  1988 


Accuracy  and  speed  (mins)  of  diagnoses  during  fault-linding  tests  for  the  experimental 

conditions. 


Fault-linding  tests 

Test  3 


Test  1 

Test  2 

•Old’ 

faults 

’New  (same  category)' 
faults 

Condition  T(FP2) 

Accuracy 

mean 

OlO 

0-20 

0-53 

0-43 

s.d. 

0-11 

0-19 

0-30 

0-20 

Speed 

mean 

3-99 

3-09 

2-22 

2-87 

s.d. 

0-85 

l>77 

0-67 

0-93 

Condition  H(FP2) 

Accuracy 

mean 

ail 

0-24 

0-53 

0-23 

s.d. 

0-13 

0-18 

0-21 

0-17 

Speed 

mean 

401 

2-70 

2^5 

2-76 

s.d. 

0-83 

0-89 

052 

0-55 

Condition  HiCDl) 

Accuracy 

mean 

0-22 

0-22 

0-38 

0-30 

s.d. 

0-18 

0-21 

0-27 

0-19 

Speed 

mean 

419 

296 

2-57 

2-97 

s.d. 

0-49 

067 

0-58 

0-41 
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Patrick  et  al./  1988 


Table  2.  Diagnostic  heuristics  for  FP2  plant. 


1.  Locution — hnd  the  general  area  of  failure  by  locating  the  disturbance  furthest  upstream. 

2.  Check  for  a  failure  in  one  of  the  pumps  which  would  produce  a  rapid  change  in  how. 

3.  Check  all  control  loops  m  the  disturbed  area  to  ensure  that  valves  are  in  the  appropriate 
positions. 

4.  If  the  failure  is  in  the  reacior/heat  exchange  area  determine  whether  it  is  either  in  the  reactor  or 
the  heat  exchange  system.  A  failure  in  the  heat  exchange  system  will  produce  symptoms  in 
column  A  but  not  in  column  B. 

5.  If  the  failure  is  in  the  feed  system,  check  whether  it  is  in  stream  X  or  Y.  If  hoih  feeds  are 
disturbed,  the  fault  is  in  feed  Y.  If  only  feed  X  is  disturbed,  the  fault  is  in  feed  X. 

6.  Check  if  duplicate  instruments  provide  the  same  readings.  If  they  do  not,  there  is  an 
instrument  reading  fault. 
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Patrick  et  al.,  1988 


Design  of  experiment. 


Experimental  conditions 


T(FP2) 

H(FP2) 

HlCDh 

Training 

1.  Module  I:  pretraining 

FP2 

FP2 

CDI 

2.  Module  2:  plant  overview  (training) 

FP2 

FP2 

CDI 

3.  Fault-finding  Test  1  (8  faults) 

FP2 

FP2 

CDI 

4.  Module  3:  training  materials 

Technical  story 

Diagnostic 

Diagnostic 

for  FP2 

heuristics 

heuristics 

5.  Fault-finding  Test  2  (8  faults) 

FP2 

for  FP2 

FP2 

for  CDI 
CDI 

6.  Training  in  fault-finding 

FP2 

FP2 

CDI 

7.  Fault-finding  Test  3 

(12  faults— -5  ‘old’.  5  ‘new  (same 
category)’  and  2  'new  (different 

category)’) 

FP2 

FP2 

CDI 

Transfer 

8.  Module  4:  plant  overview  (transfer) 

CDI 

CDI 

FP2 

9.  Fault-finding  Test  4  )K  faults) 

CDI 

CDI 

FP2 
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Rabbit,  P.,  Gumming,  G.,  and  Vyas,  S.  (1979).  Improvement, 
Learning  and  Retention  of  Skill  at  Visual  Search.  The 
Experimental  Psychology  Society.  31.  441-460. 


Task;  Visual  search 

Taxa:  Perception/visual  discrimination 

Type  of  Data;  Individual 

Subjects;  3  Experimental  psychologists 

Design;  Repeated  measures 

Training; 

1.  28  Practice  trials  on  visual  search  task 

2 .  Final  Test  -  "Transfer”  task  consisting  of 
discriminating  targets  from 

different  background 

Independent  Variable (s); 

1.  Length  of  practice  on  task  requiring  visual  search 
for  embedded  letters 

Dependent  Measure (s); 

1 .  Reaction  time  for  visual  search  of  target  letters 

Abstract;  This  experiment  examined  factors  responsible  for 
improvement  in  visual  search.  Specifically,  the  study 
investigated  the  visual  performance  of  very  highly  practiced 
subjects.  Results  demonstrated  that  after  25  days  of 
training,  variations  in  the  size  of  the  target  set  no  longer 
affected  visual  search  time.  This  may  be  explained  in  terms 
of  learning  of  specific  cue  systems  since  subjects  showed 
perfect  transfer  to  displays  on  which  they  searched  for  the 
same  target  items  among  new  background  items. 
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Rabbit  et  al.,  1979  (continued) 


Reaction  times  in  ms  for  different  subsets  of  data  from  forefinger  and  second  fint>er 


2-'rarj»ets 

4-'raruets 

O-’rarj'ets 

rorclinucrs 

I'orctinKvrs  repeated 

.152 

.15f> 

360 

One  forefint'ur  after  the  other 

3«6 

3S9 

394 

Second  finders 

Second  tinker  repeated 

369 

373 

Second  finijer  after  other  second  tinker 

390 

39H 

Second  hn^vr  after  forefintjcr  of  same  hand 

3H5 

397 

Second  Hn^cr  after  forehnKcr  of  ditferent  hand 

422 

431 

Second  hn^^er  after  third  Hnfter  of  same  hand 

4>4 

427 

Second  Hnfjcr  after  third  Anf'cr  of  different  hand 

435 

446 
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Rabbit  et  al.,  1979  (continued) 


Reaction  times  and  sigma  in  ms  for  last  5  days  of  practice;  6-target  S-lettcr  diiplav 

condition  and  after  transfer 


Day  26 

Day  27 

Day  aS 

Day  29 

Dii\  ‘{o 

Day  31 
transfer 

Mean  RT 

604 

S32 

508 

554 

55M 

Shi 

Sigma 

Sa 

54 

00 

t  i 

5'* 

Ucaction  times  and  sigmas  in  ms  for  different  fingers 


Second  I'liird 
Forvlinuer  finv'vr  Hnitcr 


a-'I'aruet  c<iiiditiiHi 

Mvjii  RT 

J»x> 

Sigma 

S» 

4  -'1  arget  ctmdition 

Mean  KT 

386 

40* 

Sigma 

54 

61 

^-I'argvc  condititHi 

Mean  RT 

jy5 

443 

436 

Sigma 

5<> 

Co 

63 
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Rabbit  et  al.,  1979  (continued) 


Reaction  times  and  sigmas  in  ms  in  each  oj  nine  conditions 


No.  of 

No.  of 

Days 

Days 

l>ays 

l>ays 

Days 

l.)ays 

target 

backcround 

»-s 

6-10 

li-iS 

i6>20 

21-25 

26-30 

symbols 

symbols 

Rl'  means 

RT  means 

HI'  means 

HT  means 

K'r  means 

HI’  means 

souijht 

on  display 

8c  si^ma 

8c  sif'ma 

8;  si^ma 

8c  siitma 

8c  si({ma 

8c  sigma 

2 

3 

632 

S'5 

486 

424 

404 
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a  131 

0  1 1 1 
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n  81 
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702 
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44  ‘ 

n  14V 

a  123 

0  112 

a  02 

a  QO 

(1  56 

U 
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6vv7S 
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n  108 

n  134 

a  122 

a  00 

a  o> 

a  81 

4 

3 

873 

620 

5“ 

453 

42b 

304 

0  212 

0  151 

0  90 

a  104 

0  66 

0  5« 

5 

1021 

712 

582 

407 

45« 

430 

a  20V 

n  iSi 

0  104 

0  100 

0  52 

0  52 

8 

1240 

935 

648 

564 

507 

400 

<T  224 

0  124 

a  114 

0  78 

ft  64 

*T  54 

6 

3 

y'4 

707 

673 

520 

487 

426 

a  2iK 

n  1H4 

a  104 

0  08 

n  72 

a  60 

5 

lobi 
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694 
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5»4 

501 

a  251 

n  IV2 

a  114 

fj  08 

74 

a  60 

K 
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997 

776 

618 

503 

560 
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n  108 
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Rabbit  et  al.,  1979  (continued) 


Reaction  times  and  sigmas  in  ms 
First  Runs 

No.  of  tarirets  for  which 
search  ^  conducted 


No.  of  symbols - 

on  display  249 


Subject 

2 

704 

741 

918 

G.D.C. 

98 

98 

1 18 

4 

811 

894 

1123 

115 

1 1 1 

128 

9 

1108 

12S3 

1412 

142 

156 

159 

Subject 

2 

7^3 

842 

987 

S.M.V. 

108 

104 

1 18 

4 

887 

1008 

1113 

117 

1  1  2 

iSo 

9 

1180 

1367 

1603 

156 

171 

214 

Subject 

2 

631 

796 

894 

B.R. 

62 

I  I  I 

124 

4 

741 

9()2 

1121 

1x7 

124 

137 

9 

1026 

1280 

146b 

136 

X49 

154 
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and  last  runs 


Last  Runs 

No.  of  tariiets  for  which 
search  was  conducted 

No.  of  symbols - 

on  display  249 


508 

529 

559 

48 

54 

41 

540 

538 

557 

58 

56 

54 

614 

5b9 

591 

62 

60 

64 

373 

473 

450 

47 

50 

.50 

497 

445 

497 

52 

49 

48 

599 

672 

774 

55 

52 

5  I 

461 

447 

40 1 

36 

37 

29 

463 

467 

401 

36 

49 

32 

541 

626 

540 

38 

47 

42 
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Rabbit  et  al.,  1979  (continued) 


Reaction  times  and  sigmas  in  ms  and  f^ercentagc  of  errors  made  in  £^/*crimcnt  I 


First  practice 
session 

Last 

practice 

session 

Recall 
session 
(Rc-test  or 
transfer) 

2  weeks  delay 

Retest  croup 

Mean  RT 

1  io8 

1S96 

899 

bet\veen  practice 

(lo  subjects) 

(<5  149) 

(-7  131) 

(7  I  ^z) 

and  recall 

"A,  Krrors 

2-4% 

fi'X. 

Transfer  croup 

Mean  RT 

10H6 

909 

946 

(lo  subjects) 

{•s  154) 

(■7  1 14) 

(7  144) 

%  Krrors 

1-9% 

21X. 

1-9% 

4  weeks  delay 

Retest  croup 

Mean  RT 

1004 

885 

S07 

between  practice 

(lo  subjects) 

(^T  162) 

(7  128) 

(7  137) 

and  recall 

%  Ivrrors 

1-6% 

2+'X 

2-7‘/, 

Transfer  croup 

Mean  RT 

1 1 14 

882 

05^ 

(lo  subjects) 

in  186) 

(7  132) 

(7  159) 

Krrors 

I'S-y, 

i')'K 

6  weeks  delay 

Retest  group 

Mean  RT 

1081 

914 

948 

between  practice 

(lo  subjects) 

(9  134) 

(7 148) 

(7  169) 

and  recall 

%  Errors 

1-2% 

2- 1'/, 

yiX 

Transfer  group 

Mean  RT 

1104 

894 

97 1 

(lo  subjects) 

(1  14^) 

(7  136) 

(7  182) 

%  Errors 

i-9% 

1-9% 

2-8'% 
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Roberts,  T.L.,  and  Moran,  T.P.  (1983).  The  evaluation  of  text 

editors:  Methodology  and  empirical  results.  Communications 
of  the  ACM.  26.  265-283. 


Task;  Edit  text 

Taxa:  Motor;  Information  processing/Problem  solving 

Subjects:  4  Novice  computer  users 

Training: 

1.  Each  subject  trained  on  9  different  text  editors 
Independent  Variable (s)  : 

1 .  Experience  with  text  editors ;  Novice  vs  Expert 
Dependent  Measure (s) ; 

1.  Time-score  for  each  text  editor 

2.  Individual  error  score  for  each  text  editor 

Abstract:  This  study  presents  a  methodology  for  evaluating 

text  editors  on  the  following  dimensions:  time  required  for 
experts  to  perform  basic  editing  tasks,  time  used  by 
experts  to  make  and  correct  errors,  rate  of  novice  learning 
of  basic  text  editing  tasks,  and  fTjnctionality  of  editors 
over  more  complex  tasks.  Time,  errors,  and  learning  are 
measured  experimentally;  functionality  is  measured 
analytically.  Nine  diverse  editors  were  evaluated  producing 
an  initial  database  of  performance  results.  The  database 
not  only  provides  information  pertaining  to  the  editors,  but 
also  provides  useful  information  about  the  users  -  the 
magnitude  of  individual  differences  and  factors  affecting 
novice  learning. 
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Roberts  et  al . , 


1983  (continued) 
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Roberts  et  al.,  1983  (continued) 


Av«r«9«  LMming 
Curvn  ovtr  all  Laamara  on  aacti 
Editor.  Tha  two  teco  curvaa  wara 
producad  by  diffarant  inutnietota. 
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Expert  Error  Time  Expert  Error-Free  Performarice  Time 
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Roberts  et  al.,  1983  (continued) 
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Rouse,  W.  B.  (1978) .  Human  problem  solving  performance  in  a 
fault  diagnosis  task.  IEEE  Transactions  on  Systems.  Man, 
and  Cybernetics.  SMC- 8.  258-271. 


Task:  Fault  diagnosis  in  graphically  displayed  network 
problems 

Taxa;  Problem  solving 
Type  of  Data:  Group 

Subjects:  8  Engineering  graduate  students 
Training: 

1 .  Trained  with  and  without  display  aiding 

2.  Final  Test  “  "Transfer"  session  of  either  aided  or 
unaided  displays  depending  on  training 

Independent  Variable (s) : 

1 .  Problem  size  in  fault  diagnosis  using  graphically 
displayed  network  system  as  simulated  diagnostic  task 

2.  Pacing  (forced  vs  non-forced) 

3.  Display  (computer  aided  vs  non-aided) 

Dependent  Measure (s): 

1 .  Number  of  tests  until  correct  diagnosis 

2.  Percent  correct 

Abstract;  This  study  examined  the  design  of  visual  displays 
used  for  problem  solving  tasks.  Two  experiments  were 
conducted  investigating  human  problem  solving  performance  in 
the  diagnosis  of  faults  in  graphically  displayed  network 
problems.  The  effects  of  problem  size,  forced-pacing, 
computer  aiding,  and  training  were  considered.  The  results 
indicated  that  human  performance  deviates  from  optimum  as 
problem  size  increases.  It  appears  that  forced-pacing 
causes  the  human  to  adopt  brute  force  strategies  compared 
with  strategies  adopted  during  self-paced  situations. 
Computer  aiding  greatly  lessens  the  number  of  mistaken 
diagnoses  by  performing  the  bookkeeping  portions  of  the 
task. 
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Average  ^iimber  of  Tests  Until  Correct  Solution  (Subject-Optimal) 


Rouse/  1978  (continued) 
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Average  number  of  tests  until  correct  solution  -  transfer  data- 

experiment  one. 
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Rouse/  1978  (continued) 


Analysis  of  Average  Number  of  Tests  Until  Correct  Solution 


(Subject-Optimal)  -  Training  Data  -  Experiment  One 


SOURCE 

SS 

df 

Var. 

F 

Between-Sub jects  Factors: 

■ 

Displays 

2.258 

H 

2.258 

2.12 

S  Nested  with  Displays 

6.128 

6 

1.021 

Within-Sub jects  Factors: 

Practice 

0.414 

1 

0.414 

2.65 

Practice  x  Displays 

0.008 

1 

0.008 

.05 

Practice  x  S  Nested  with 
Displays 

0.935 

6 

0.156 

Size 

1.825 

2 

0.913 

11.16 

Size  X  Displays 

0.433 

2 

0.217 

2.12 

Size  X  S  Nested  with 
Displays 

0.920 

12 

0.077 

Practice  x  Size 

1.806 

2 

0.903 

11.16 

Practice  x  Size  x  Displays 

0.015 

2 

0.008 

0.10 

Practice  x  Size  x  S  Nested 
With  Displays 

0.984 

12 

0.082 
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Schneider,  W.,  &  Fisk  (1984).  Automatic  category  search  and  its 
transfer.  Journal  of  Experimental  Psychology:  Learning. 
Memory,  and  Cognition.  10.  1-15. 


Task:  Visual  search 

Taxa:  Visual/Information  processing 

Type  of  Data:  Group 

Subjects:  Experiment  1:  20  Introductory  Psychology  students 

Experiment  2 :  6  Introductory  Psychology  students 

Design:  Exp.  1:  within-subjects,  no  repeated  measures 

Exp.  2:  Within-sub jectS/  no  repeated  measures 


Training: 

1.  Exp.  1:  Subjects  were  provided  different  number  of 
category  exemplars  previous  to  a  visual  search  task 

2.  Exp.  2:  Subjects  were  trained  to  detect 
consistently  mapped  category  words  while  controlled 
processing  resources  were  allocated  to  a  variably 
mapped  digit  task 

Independent  Variable (s): 

1.  Exp.  1:  Type  of  mapping  (consistently  mapped  or 
variably  mapped)  used  for  training  of  category  search 
task 

2.  Exp.  2:  Number  of  concurrent  tasks  (single 
or  dual  task) 

Dependent  Measure (s) ; 

1.  Exp.  1:  Visual  search  reaction  time  improvement 

2.  Exp.  2:  Percent  improvement  of  word  detection 

Abstract:  Experiments  examined  practice  and  transfer 
effects  in  consistently  mapped  (CM)  and  variably  mapped  (VM) 
semantic  word  search  tasks.  Experiment  1  examined 
improvements  in  reaction  time  in  detecting  words  from  a 
category  as  a  function  of  the  number  of  exemplars  in  the 
category.  All  CM  conditions  showed  improvement.  Experiment 
2  showed  that  practice  reduced  resource  sensitivity  in  CM 
category  but  did  not  benefit  VM  category  search. 
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DETECTION  ACCURACY 


REACTION  TIME 


Schneider  et  al.,  1984 


Exparlintnt  la:  Raaetlen  tima  aa  a  function  of  praetlea.  (CM4,  CMA.  and  CM13  rotor  to  training 
category  oat  alzaa  of  4,  i,  and  13  raapactlvaly:  laactlon  lima  la  In  fflllllaaconda.  Each  point  ropraaanta 
38  triala  par  aubjaet.  CM  ■  conalatant  mapping:  VM  <•  varied  mapping.) 
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Scott/  P.G.,  McDaniel/  W.C./  &  Braby/  R.  (1982) .  Improved 
procedures  training  through  use  of  aids  developed  from 
learning  guidelines  (Navy  Technical  Report  No.  113) . 
OrlandO/  FI;  Department  of  the  Navy. 


Taslc:  Operate  a  cockpit  procedures  trainer  (CPT) 

Taxa:  Information  processing/Motor 

Type  of  Data:  Group 

Subjects:  35  newly  designated  Naval  Aviators 
Design:  Between  subjects/  no  repeated  measures 

Training: 

1.  Trained  with  traditional  procedures  or  with 
supplemental  training  materials . 

Independent  Variable (s): 

1,  SH-3  Aircraft  Training  aids;  traditional  or 
supplemental 

Dependent  Measure (s): 

1.  Errors  on  the  cockpit  procedures  trainer  (CPT) 

Abstract;  This  report  describes  the  development  of  a 
procedural  aid  for  the  SH-3  Aircraft  Normal  Start  Checklist. 
Subjects  either  received  the  training  aid  or  just  the 
traditional  instruction  for  the  SH-3  Aircraft.  The  first 
measure  taken  examined  students  performance  on  their  initial 
attempt  to  accomplish  the  engine  start  in  the  CPT  (Device 
2C44) .  The  second  measure  examined  each  students  sequence 
of  graded  trials  in  order  to  determine  the  point  at  which 
the  normal  start  task  was  performed  reliably  to  NATOPS 
standards.  Results  indicated  that  subjects  receiving  the 
procedural  aid  performed  the  initial  start  significantly 
better  than  subjects  who  received  traditional  instruction. 
The  second  measure  indicated  that  subjects  receiving  the 
procedural  aid  demonstrated  proficiency  significantly  sooner 
than  those  who  received  only  traditional  instruction. 
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Scott  et  al.,  1982  (continued) 
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CUMULATIVE  %  ATTAININQ  PROFICIENCY 


Scott  et  al.,  1982  (continued) 
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Results  of  Engine  Start  Field  Test  Showing  Cumulative  Subject 
•  Proficiency  Attainment  for  Two  Types  of  Training  Materials 


Sheppard  D.J.  (1984)  .  Visual  and  part-tas)c  manipulations  for 
teaching  simulated  carrier  landings  (NAVTRAEQUIPCEN  81-C- 
0105-9)  .  Westlalce  Village,  CA:  Canyon  Research  Group,  Inc 
(DTIC  No.  AD-A153622) . 


Tas)c:  Simulated  aircraft  carrier  landing 

Taxa:  Motor/ Information  processing 

Type  of  Data:  Group 

Subjects:  36  males  college  students  with  no  flight 

experience 

Design:  2  (whole  tas)c  or  part  task  training)  x  2  (small  or 
large  Fresnel  Lens  Optical  Landing  System  (FLOLS) )  x 
3 (conventional,  rate  or  command  display)  factorial  design 


Training: 

1.  Subjects  were  trained  under  either  the  part -task 
condition  or  the  whole-task  condition. 

2.  Subjects  were  trained  using  either  a  small  or 
large  FLOLS. 

3.  Subjects  were  trained  using  either  a  conventional, 
rate,  or  command  display. 

Independent  Variable (s): 

1.  Task  configuration:  part  task  vs  whole  task 

2.  FLOLS  type:  conventional  or  vertical  bars  added 

3.  FLOLS  size:  normal  size,  1.5  times  normal  size, 
4.5  times  normal  size 

Dependent  Measure (s): 

1.  RMS  Glideslope  Error 

Abstract:  This  study  investigated  a  segmentation  method  of 
part-task  training  and  two  types  of  visual  augmentation  for 
teaching  simulated  carrier  landings.  The  two  visual 
enhancements  included  a)  adding  two  types  of  descent  rate 
information  and  b)  enlarging  the  FLOLS  display.  The 
sequence  consisted  of  30  training  trials  with  instructional 
feedback  under  a  particular  condition  followed  by  30  test 
trials  with  no  instructional  feedback  under  the  criterion 
condition.  Most  importantly,  transfer  from  a  large  display 
to  a  large  display  had  no  detrimental  effects. 
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Sheppard, 


1984  (continued) 
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error  during  transfer. 
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Sheppard, 


1984  (continued) 
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Block  X  task  interactions  for  RM3  giideslooe 
error  during  transfer. 


A-96 


Simon,  C.W.,  &  Roscoe,  S.N.  <1981).  Application  of  a  multi¬ 
factor  approach  to  transfer  of  training  research 
(NAVTRAEQUIPCEN  78-C-0060-6) .  Westlake  Village,  CA:  Canyon 
Research  Group  (AD-A108499) . 


Task:  Tracking  of  symbols  on  a  plasma-panel  display 

Taxa;  Visual 

Type  of  Data:  Group 

Subjects:  80  right-handed  male  non-pilots 

Design:  3  (control  order  %  acceleration)  x  3 (display  lag)  x 
3  (tracking  mode  %  pursuit)  x  3 (prediction  time)  x  3 (control 
gain)  x  3  (numbter  of  training  trials)  factorial  design 

Training: 

1.  Subjects  were  trained  on  an  aircraft  simulator 
under  one  of  the  above  conditions 

Independent  Variable  (s): 

1.  Simulator  configurations  (variations  of  the 
following) :  Control  order,  display  lag,  tracking  mode, 
prediction  time,  control  gain,  and  number  of  trials 

Dependent  Measures) : 

1.  Tracking  error  as  measured  on  the  Visual  Technology 
Research  Simulator 

Abstract:  This  study  was  conducted  to  establish 
relationships  among  training,  test,  and  transfer  scores 
related  to  the  maneuvering  of  a  vehicle,  and  to  determine 
the  relative  complexities  of  response  surfaces.  Also,  this 
study  demonstrated  a  new  transfer  research  paradigm,  and 
extracted  from  the  available  data  information  that  enable 
the  estimation  of  cost  of  the  effectiveness  of  simulator 
characteristics.  A  horizontal  tracking  task  was  used  in 
this  study.  Six  factors  were  manipulated  including  vehicle 
control  order,  display  lag,  tracking  mode,  prediction  time, 
control  gain,  and  number  of  training  trials.  Results 
indicated  that  the  transfer  surface  appeared  less  complex 
than  the  training  surface,  the  relationship  between  training 
and  transfer  variables  was  positive  but  weak,  some  factors 
had  strong  effects  in  training  and  weak  effects  in  transfer 
and  vice  versa,  and  transfer  was  facilitated  when  the  values 
of  certain  variables  resulted  in  training  conditions  that 
were  more  difficult  than  the  transfer  criterion  conditions. 
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Simon  et  al.^  1981  {continued) 


Critical  ifiteractlons  b«tween  equipment/training  factors 
and  ptiases. 
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Weitzman,  D.O.,  Fineberg,  M.L.,  Gade,  P.A.,  &  Compton,  G.L 
(1979) .  Proficiency  maintenance  and  assessment  in  an 
instrument  flight  simulator.  Human  Factors.  21.  701-710. 


Task:  To  pilot  helicopter  simulator  (Device  2024) 

Taxa:  Visual/Motor/Information  processing 
Type  of  Data:  Group 

Subjects:  36  fully  qualified  combat-ready  Army  pilots 

Design:  Repeated  measures 

Training: 

1.  Subjects  were  trained  either  by  aircraft,  by 
simulator,  or  by  both  aircraft  and  simulator. 

Independent  Variable (s): 

1.  Training  instrument:  Device  2B24  simulator,  or 
the  UH-IH  helicopter,  or  both 

Dependent  Measure (s) : 

1.  Instructor  helicopter  pilot  ratings 

Abstract:  This  study  evaluated  transfer  effects  of 
experienced  Army  pilots  on  a  high  fidelity  flight  simulator 
(Device  2B24) .  Evidence  that  supported  positive  transfer 
was  obtained  by  comparing  pilots  trained  in  the  simulator 
with  pilots  trained  in  the  aircraft  (UH-IH)  and  with  pilots 
trained  in  both.  Also,  training  in  the  simulator  accurately 
predicted  performance  in  the  aircraft.  This  suggests  that 
simulators  of  proven  effectiveness  can  be  used  both  to 
maintain  and  assess  the  proficiency  of  experienced  pilots. 
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Weitzman  et  al.,  1979  (continued) 


Uaminit  curves  of  ehtekride  performance 
for  the  fMO  pilot  groups  and  three  training  modes. 
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Weltzman  et  al./  1979  (continued) 


OlVICi  2124  TISI  SCOItS  (i) 

Inflight  test  satres  plotted  against 
simulator  test  scores  for  the  last  test  period.  Solid  line: 
least  squares,  best-fit  line. 
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Wightman,  D.C.,  &  Sistrunk,  F.  (1987).  Part-task  training 
strategies  in  simulated  carrier  landing  final  approach 
training.  Human  Factors.  29.  245-254. 


Task:  Simulated  carrier  landing 

Taxa;  Motor/Visual 

Type  of  Data:  Group 

Subjects:  40  male  undergraduates 

Design:  Repeated  measures 

Training: 

1.  Subjects  were  trained  on  a  simulator  under 
conditions  of  whole  task  with  normal  lag,  whole  task 
with  progressively  increased  lag,  segmented  task  with 
normal  lag,  or  segmented  task  with  progressively 
increased  lag. 

Independent  Variable (s) : 

1.  Training  task:  whole  task  vs  segmented  task 

2.  Carrier  landing  lag:  normal  lag  vs  progressive 
lag 

Dependent  Measure (s): 

1.  Root  mean  squared  error  (RMS)  carrier  landing  as 
measured  on  the  Visual  Technology  Research  Simulator 

Abstract:  This  study  examined  part -task  training  strategies 
on  transfer  to  simulator  carrier  landing.  College  students 
were  taught  carrier  landing  final  approach  skills  in  a 
simulator  under  one  of  three  experimental  conditions,  and 
were  then  tested  in  the  simulator  on  a  criterion 
configuration  that  was  identical  to  the  control  training 
condition.  Also,  subjects  motor-skill  aptitude  was 
assessed.  Results  demonstrated  that  training  under  backward 
procedures  of  chaining  produced  better  transfer  to  criterion 
than  did  an  equal  number  of  training  trials  on  the  criterion 
task  itself.  An  interaction  of  condition  and  aptitude 
revealed  that  the  chaining  method  was  particularly  effective 
for  low  aptitude  subjects. 
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Trclnlng  Conditlont 


1.2  3.«  7.1  11.14 

TRANSFER  TRIALS 


Qrtph  of  Bloeks  x  Taik  x  Motor  Skill  Intorootlon  for  tranofor  trial  blocks  1000  foot  to  tho  ramp. 


A-104 


Wrisberg,  C.A.,  &  Winter,  T.P.  (1983).  Variability  of  practice 
and  the  transfer  of  training  of  motor  skills  (ARI  Technical 
Report  596).  Alexandria,  VA:  U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences. 


Task:  Visual  tracking 
Taxa:  Visual/Motor 
Type  of  Data:  Group 

Subji^dCts:  288  right-handed  students  at  the  University  of 

Tennessee 

Design:  Repeated  measures 

« 

Training: 

1.  Subjects  were  trained  under  either  a  closed  skill 
condition  or  an  open  skill  condition. 

Independent  Variable  (s): 

1.  Training  condition:  open  skill  (requires  varying 
reaction) ,  closed  skill  (requires  stable  reaction) 

Dependent  Measure  (s): 

1.  Absolute  error  (msec)  of  arm  movement  during 
transfer 

Abstract:  This  study  examined  whether  the  amount  and 

variability  of  practice  enhanced  transfer  to  non-practiced 
tasks  of  a  similar  nature.  Siibjects  were  trained  in  either 
an  open  skill  or  open  skill  situation.  Subjects  in  the  open 
skill  situation  visually  tracked  a  moving  light  sequence  and 
attempted  to  complete  a  ballistic  arm  movement  coincident 
with  the  termination  of  the  light  movement.  Those  in  the 
closed  skill  condition  simply  focused  their  attention  on  the 
production  of  a  specific  movement  on  each  trial.  Variations 
in  movement  time  and  movement  distance  resulted  in  higher 
error  for  those  in  the  closed  skill  situation  compared  with 
subjects  in  the  open  skill  condition.  However,  those  in  the 
open  skill  condition  demonstrated  transfer  performance  that 
was  less  accurate  and  less  consistent  than  that  of  subjects 
in  the  closed  skill  condition. 
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TRAINING  BLOCK 


flean  Absolute  Error  During  the  Training  Phase  for  the 
Movement  Distance  x  Movement  Time  x  Blocks  Interaction  for 
Closed  Skill  Subjects  Receiving  4S  Trials  (9  Blocks) 
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Mean  Constant  Error  During  (Tpen  SkiU  Transfer  for  the 
Interaction  Between  Number  of  Training  Trials  and 
Stimulus  Velocity  Condition  During  Training 
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APPENDIX  B 


REFERENCES  FOR  RELEVANT  RESEARCH  ARTICLES 


The  purpose  of  Appendix  B  is  to  provide  a  bibliography  of 
articles  obtained  during  the  searches  that  are  considered  to  be 
relevant  to  the  project.  Relevancy  of  these  publications  is 
defined  in  terms  of  several  criteria  including:  presence  of 
quantitative  data,  appropriate  statistics  and  design,  use  of  an 
adult  human  population,  use  of  predictor  and  criterion  variables, 
and  use  of  a  task  type  compatible  with  one  of  the  nine  taxa. 
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APPENDIX  C 


ANNOTATED  BIBLIOGRAPHY  OF  RESEARCH  REVIEWED  FOR  THIS  PROJECT 


The  purpose  of  Appendix  C  is  to  provide  a  complete 
bibliography  of  the  publications  that  were  reviewed  during 
searches  for  relevant  literature.  This  appendix  contains  both 
the  publications  that  were  reviewed  and  at  some  point  rejected 
and  those  that  were  selected  as  relevant  to  the  study. 
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APPENDIX  D 


ADDDITIONAL  DESCRIPTION  OF  LEARNING  CURVE  EQUATIONS 


The  purpose  of  Appendix  D  is  to  provide  equations  and 
derivatives  used  by  prominent  human  learning  theorists.  This 
appendix  may  be  used  in  conjunction  with  textual  discussions  of 
the  implementation  of  learning  curves  to  training  and  performance 
issues.  Appendix  D  may  also  provide  a  more  detailed  view  of 
learning  curve  equations  than  is  provided  in  the  text. 
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LEARNING  CURVE  EQUATIONS 


Thurstone^s  Learning  Curve  Equation 

After  experimenting  with  about  forty  different  equations  on 
published  learning  curves,  Thurstone  selected  a  form  of  the 
hyperbola  taking  the  form; 

X*R 

in  which  Y  =  attainment  in  terms  of  the  number  of  successful 
acts  per  unit  time. 

X  =  formal  practice  in  terms  of  the  total  number  of 
practice  acts  since  the  start  of  formal  practice. 

L  =  limit  of  practice  in  terms  of  attainment  units. 

R  =  rate  of  learning  which  indicates  the  relative 
rapidity  with  which  the  limit  of  practice  is  being 
approached.  It  is  numerically  high  for  a  low  rate  of 
approach  and  low  for  a  high  rate  of  approach. 

The  above  equation  represents  a  learning  curve  which  passes 
through  the  origin,  i.e.,  starting  with  a  zero  score  at  zero 
formal  practice.  However,  most  learning  curves  start  with  some 
finite  score  even  at  the  initial  performance.  For  learning 
curves  that  do  not  pass  through  the  origin,  the  equation  is: 

{X*P)  *R 


in  which  P  =  equivalent  previous  practice  in  terms  of  formal 
practice  units. 

Thus,  the  first  equation  can  be  rectified  as  follows: 


y_  L*X 
■  X+R 


XY^RY=LX 
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The  above  equation  is  linear  if  X/Y  is  plotted  against  X. 
Similarly,  the  second  equation  may  be  rectified  if  written  in  the 
following  form; 


X+  (R+P) 


which  becomes  linear  when  (X  +  P)/Y  is  plotted  against  X.  When 
so  rectified,  the  constants  L,  R,  and  P  may  be  determined  by 
several  methods.  The  choice  of  methods  depends  on  the  scatter  of 
the  data,  the  desired  accuracy,  and  the  number  of  curves  one  must 
calculate . 


Thurstone^s  Learning  Function 

The  two  principle  variables  in  learning  are:  a)  practice 
and  b)  attainment.  Attainment  is  an  increasing  function  of 
practice.  Practice  may  be  measured  either  in  terms  of  repetition 
or  in  terms  of  time  devoted  to  it  (effort  is  assumed  to  be 
constant) .  Repetitions  may  be  counted  as  successful  acts  only, 
or  both  successes  and  failures.  Attainment  can  also  be  expressed 
in  various  ways,  but  for  the  present  purpose  it  will  be  expressed 
in  terms  of  the  probability  that  an  act  will  be  counted  as 
successful.  This  measure  of  attainment  is  proportional  to  the 
number  of  successful  acts  per  unit  time.  An  additional 
assumption  is  that  the  rate  at  which  acts  are  initiated  is 
directly  proportional  to  effort  (which  is  assumed  to  be 
constant).  Thus  far,  classifications  are  as  follows: 

s  =  total  number  of  acts  that  the  learner  is  liJcely  to 
initiate  and  that  would  be  credited  as  successes. 

e  =  total  number  of  acts  that  the  learner  is  li)cely  to 
initiate  and  that  would  lead  to  error  or  failure  or  loss  of 
time. 

p  *  probability  that  the  act  initiate  at  any  moment  of  time 
will  lead  to  successful  completion. 

q  -  probability  that  the  act  initiated  at  any  moment  of  time 
will  lead  to  error  or  loss  of  time. 

Thus,  clearly: 
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<b) 


e 

s+e 


and 


(c) 


ps*pe»a 


The  object  of  the  learner  is  to  reduce  the  errors  e  and  to 
raise  the  proportion  p. 

Analytically/  this  can  be  approached  as  a  probability  issue. 
For  the  sa]ce  of  demonstration/  let  all  probable  or  plausible  acts 
of  a  learner  be  represented  by  white  and  black  balls  in  a  basket. 
At  any  moment  while  the  learner  is  drawing  them  out/  there  is  a 
certain  probability  p  that  the  ball  that  is  drawn  out  will  be 
white  and  a  certain  probability  q  that  it  will  be  black.  If  it 
turns  out  to  be  black/  then  there  is  the  probability  k  that  it 
will  be  eliminated  from  the  basket  and  the  probability  (1-k)  that 
the  ball  will  be  returned  to  the  basket. 


p  =  probability  that  a  white  ball  will  be  drawn  (success) 
q  =  (1-p)  =  probability  that  a  black  ball  will  be  drawn 
(error) 

kq  »  probability  that  a  black  ball  will  be  drawn  and 
eliminated  from  the  basket 

q(l-k)  =  probability  that  a  black  ball  will  be  drawn  and 
that  this  error  will  be  repeated  in  future  attempts 

However/  when  an  error  is  eliminated/  obviously  the  total 
number  of  possible  errors  e  is  reduced  by  one.  AlsO/  if  the 
probability  that  a  particular  draw  will  give  a  black  ball  and 
that  it  will  be  eliminated  is  kq/  then  this  is  also  the  average 
number  of  black  balls  by  which  e  is  reduced  per  draw.  ThuS/  we 
are  able  to  write  the  following  differential  equation: 


(d) 


de 

dt 


by  the  earlier  equation  (b)  we  have: 


de_  -kB 

dt  a*  9 
(e) 

So  far,  there  has  been  no  mention  of  the  possibility  that  a 
successful  act  has  some  effect  favoring  its  recurrence.  We  will 
assume  not  only  that  overly  committed  errors  decrease  their 
probability  of  recurrence,  but  also  that  overtly  successful  acts 
increase  their  probability  of  recurrence.  Thus,  following 
equation  (e) ,  we  can  express  a  differential  equation  for 
successes : 
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(f) 


da 

dt 


+kp 


which  by  equation  (a)  becomes: 


dg_  ka 
dt  a+a 

<g) 

Here  the  same  constant  k  is  used  as  was  used  in  equation 
(e) .  We  can  more  or  less  assume  that  the  extent  to  which  a 
subject  profits  from  his/her  errors  is  about  the  same  as  the 
extent  to  which  he/she  profits  from  his/her  successes.  Also,  we 
will  treat  the  total  number  of  plausible  errors  e  as  a  decreasing 
quantity,  and  the  total  number  of  plausible  successes  as  if  it 
were  an  increasing  quantity.  The  differential  equations  (e)  and 
(g)  allow  us  to  find  a  simple  relation  between  the  total  number 
of  plausible  error  e  and  effective  number  of  successes  s  which 
the  learner  has  at  his/her  disposal  at  any  particular  moment  of 
practice.  Dividing  (g)  by  (e)  we  have: 

da  _  a 


and  consequently: 

(i) 

so  that : 


logg-t-logesiog/n 

where  m  is  a  constant.  This  may  also  be  written  simply  as: 


a0=m 

<k) 

In  effect,  the  product  of  the  total  number  of  errors  and  the 
effective  number  of  successes  in  the  learner' s  repertoire  is  e 
constant.  The  physical  meaning  of  this  constant  m  is  apparently 
related  to  the  complexity  of  the  learning  task.  For  instance, 
when  the  learning  process  is  intrinsically  complex  for  the 
subject,  the  product  se  is  large  and  vice  versa. 

The  constant  k  is  an  attribute  of  the  individual  learner 
while  the  constant  m  is  primarily  an  attribute  of  the  learning 
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task.  If  we  want  to  know  the  relation  between  practice  time  t 
and  attainment  p,  from  equation  (c)  we  can  write  the  following 
differential  equation: 


(1) 


de 


ds>  _  ds 
dt  ~  dt 


Substituting  equations  (e)  and  <g)  in  (1)  we  have: 


3+e  dt  a+e  dt  a+e 
(m) 

and  collecting  terms  it  becomes: 


<  s+e)  *  -k  (p©+  a-pa) 


Thus,  the  relation  between  p  and  t  is: 


J  a-pa+pe  J 

<o) 

In  performing  the  integration,  the  variables  s  and  e  may  be 
stated  first  in  terms  of  the  individual  learning  constants  k  and 
m  and  the  desired  variables  p  and  t.  Following  from  equation 
(c) ,  this  can  be  done  as  follows: 


<P) 


e=.£l££ 

P 


and  from  equation  (k)  we  have: 


a 

(q) 

Now,  solving  equations  (p)  and  (q)  simultaneously  for  s  and 
e,  we  get: 


and: 


(r-1) 


a- 


N 


1-p 
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/n(l-p) 

P 

{r-^} 

The  numerator  of  equation  (o)  contains  the  factor  (s  +•  e)^ 
which  is  evaluated  in  terms  of  equations  (r-1)  and  (r-2)  as 
follows : 


(s) 

which  simplifies  as  follows: 


(t) 


(a+e)  2= 


m 

P(l-P) 


The  denominator  of  equation  (o)  can  also  be  evaluated  in 
terms  of  individual  learning  constants  k  and  m  and  the  desired 
variables  p  and  t: 


(u) 


s-ps+pe— 

yr:p  ^  sfp 


which  simplifies  as: 


s-ps+po=2V/np  ( 1-p) 

(V) 

Equation  <o)  can  be  rewritten  in  a  more  convenient  form  by 
substituting  equations  (t)  and  (v)  and  simplifying; 


(w) 


/ 


^  4P 
P"*  (1-P) 


3 

2 


Thus,  we  have  a  relation  between  the  two  variables  p  and  t 
of  the  learning  curve  and  the  two  constants  k  and  m. 

In  order  to  facilitate  the  integration  process,  one  may  make 
the  substitution  p  =  sin*^.  The  limits  of  p  and  sin*4>  ci^e  the 
same.  Therefore,  equation  <w)  becomes; 


(X) 


/ 


3in^^cos*4> 
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J  csc^^aec^^d^ 

■  J  (2+tan*^+cot*^>  di^ 


(y) 


s  tan4>-co  t<t>+ C= 

\fm 


But  from  the  substitution  it  follows  that: 


-jE_ 

1-p 


and: 


cot^= 


IzR 

p 


Hence  equation  (w)  becomes: 


(z) 


2p.rl 

Vp-p* 


in  which  2  is  a  constant  of  integration.  Finally,  this  is 
the  desired  learning  curve  equation. 


Generalized  Power  Function  (see  Lane,  1986) 

In  its  most  complete  version,  the  generalized  power  function 
appears  as  follows: 


(a) 


T=A+S(Ar+E) -■* 


For  the  sake  of  simplicity,  all  of  the  following  equations 
will  implement  the  following  notational  scheme: 
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N  »  number  of  trials,  time  units  elapsed  or  other  index 
of  cumulative  exposure  or  practice 

T  =  time  to  respond,  to  perform  a  task,  to  complete  one 
unit  of  output  on  a  given  trial,  or  other  measure  on 
which  improved  performance  is  indicated  by  decreasing 
values . 

Y  ==  performance  score,  output,  ratings,  or  other 
measure  on  which  improved  performance  is  indicated  by 
increasing  values. 

A  a  asymptote.  Best  possible  level  of  performance. 

The  value  of  T  or  Y  as  N  approaches  infinity. 

B  =  performance  on  first  trial  or  output  on  first 
trial. 

E  =»  prior  learning  (in  trial  equivalents)  .  Reflects 
transfer  from  prior  experience  or  learning  in  terms  of 
trials  required  to  attain  a  presumed  entry-level 
performance . 

R  =*  rate  variable  describing  amount  of  change  in  Y  or  T 
with  one  unit  change  in  N. 

The  power  function  is  also  encountered  in  a  simpler  form 
with  the  presumption  that  A  and  E  are  zero: 


(b) 


Curves  that  follow  the  power  law  show  that  the  change 
(improvement)  in  performance  between  two  trial  decreases 
systematically  as  the  number  of  trials  increases.  The  rate 
variable  R  is  a  measure  of  how  rapidly  improvement  drops  as  a 
function  of  practice  and  also  indicates  the  degree  of  curvature 
of  the  learning  curve,  the  rapidity  with  which  asymptote  is 
reached  (if  it  is  reached) .  The  decay  under  the  power  law  is 
such  that  if  T  (or  Y)  changes  by  a  given  factor  over  n  trials,  it 
will  require  another  n (n  -  1)  trials  for  T  to  change  by  that 
factor  again  (Newell  &  Rosenbloom,  1981) .  For  these  power  law 
curves  (as  well  as  most  other  curves  that  describe  learning) ,  the 
amount  of  learning  on  each  trial  is  a  constant  proportion  of  what 
remains  to  be  learned,  thus  producing  a  negatively  accelerated 
curve . 


Generalized  Exponential  (see  Lane,  1986) 

Please  refer  to  the  notational  scheme  above  as  needed.  The 
exponential  function  is  encountered  as  follows; 
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where  e  is  the  natural  logarithm.  As  later  variants  will 
show,  the  exponential  function  can  be  recast  into  a  variety  of 
forms.  The  exponential  is  substantially  different  from  the  power 
law  (Newell  &  Rosenbloom,  1981)  .  The  exponential  decreases  or 
increases  more  rapidly  since  the  amount  learned  on  each  trial 
does  not  decrease  as  a  function  of  N.  Generally,  if  T  decreases 
by  a  specific  factor  in  n  trials,  it  takes  n  additional  trials  to 
decrease  by  that  factor  again  (as  opposed  to  n(n  -  1)  trials  for 
the  power  law) . 

The  exponential  growth  curve  or  negative  exponential  usually 
takes  a  negatively  accelerated  form  similar  in  shape  to  the  power 
function.  As  shown  below,  the  function  increases  over  time: 

(b) 

where  R  has  the  conventional  meaning.  In  this  version,  the 
fit  is  based  soley  on  the  asymptote  with  use  of  initial  value. 

As  shown  below,  this  equation  is  also  sometimes  used  without  the 
prior  experience  parameter; 

(c) 

A  varying  notation  of  the  exponential  curve  employs  a 
different  interpretation  of  the  rate  variable.  As  shown  in 
equations  (d)  and  (e)  below,  the  parameter  t  is  a  rate  constant 
related  reciprocally  to  R  such  that  large  values  of  t  are 
associated  with  slower  increases  in  Y; 

r=A  [  1-©  - 


(e) 

A  version  of  the  exponential  commonly  applied  in  the 
industrial  engineering  field  is  the  "time  constant"  model 
shown  below: 


y=fi+(A-s)  (1-©-*^/*^) 


The  version  shown  above  involves  both  A  and  B  parameters 
(see  initial  notation)  and  t  is  the  time  constant.  Note  that  the 
quantity  (A  -  B)  is  the  difference  between  initial  performance 
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and  asymptote,  and  thus  represents  the  maximum  increase  in 
performance  due  to  learning.  The  time  constant  model  is 
primarily  applied  to  the  performance  of  individual  operators 
rather  than  the  total  production  system. 


Logistic  <see  Spears,  1983;  1985) 

Please  refer  to  the  notational  scheme  above  as  necessary. 
The  logistic  curve  appears  as  follows: 


y=A/ [1+ (B-A> 

<a) 

where  k  is  implicitly  a  function  of  R  such  that  k  =  R  /[Y  (A 
“  Y)],  and  e  is  the  natural  logarithm.  Note  that  in  this 
context,  although  k  is  fit  as  a  constant,  it  is  not  the  same 
constant  rate  measure  defined  for  the  other  curves.  It  is  a 
measure  that  varies  across  trials  reflecting  a  rate  that  changes 
in  accordance  with  the  amount  already  learned. 

The  logistic  curve  implements  four  parameters:  1)  Beginning 
Level,  2)  Asymptote,  3)  ,  Rate  of  Learning  and  4)  Inflection 
Point.  Beginning  level  is  defined  as  the  level  of  performance 
before  any  practice  occurs;  Asymptote  is  defined  as  the  limit  for 
performance  as  practice  approaches  infinity;  and  Rate  of  Learning 
is  defined  as  the  rate  variable  describing  the  amount  of  change 
in  performance  with  one  unit  change  in  practice  (see  notational 
scheme  above) .  Finally,  and  perhaps  most  importantly,  is  the 
Inflection  Point  component.  Spears  (1983;  1985)  places  much 
emphasis  on  inflection  points  as  important  parameters  in 
evaluating  training  progress  primarily  because  it  is  at  these 
points  where  learning  stops  increasing  and  begins  to  slow  down. 
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APPENDIX  E 


REFERENCES  FOR  SUPPLEMENTAL  LITERATURE 


The  purpose  of  Appendix  E  is  to  provide  the  reader  with 
exposure  to  additional  supporting  literature  in  areas  relevant  to 
this  project.  These  publications  have  been  categorized  according 
to  general  subject  area  for  search  convenience. 
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